{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6a764d92",
   "metadata": {},
   "source": [
    "# Gaussian Processes\n",
    "\n",
    "If one is ready to assume the data available is such that the variable being fit is Gaussian for any pair of samples taken, one can use fit the data and obtain a reliable uncertainty with a very small set of assumptions.\n",
    "\n",
    "For this example, we will use the dataset collected in Mauna Loa showing the amount of CO2 present in the atmosphere. This data may be downloaded from https://gml.noaa.gov/ccgg/trends/data.html and it should already be available in this repository.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bea9d764-7713-4d25-91bd-b2322960a30e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!wget https://raw.githubusercontent.com/European-XFEL/ml-lecture-oct-2023/main/co2_weekly_mlo.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8131588",
   "metadata": {},
   "source": [
    "We can take a quick look at the file to see what is inside:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "07ecc0d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# --------------------------------------------------------------------\n",
      "# USE OF NOAA GML DATA\n",
      "# \n",
      "# These data are made freely available to the public and the\n",
      "# scientific community in the belief that their wide dissemination\n",
      "# will lead to greater understanding and new scientific insights.\n",
      "# The availability of these data does not constitute publication\n",
      "# of the data.  NOAA relies on the ethics and integrity of the user to\n",
      "# ensure that GML receives fair credit for their work.  If the data \n",
      "# are obtained for potential use in a publication or presentation, \n",
      "# GML should be informed at the outset of the nature of this work.  \n",
      "# If the GML data are essential to the work, or if an important \n",
      "# result or conclusion depends on the GML data, co-authorship\n",
      "# may be appropriate.  This should be discussed at an early stage in\n",
      "# the work.  Manuscripts using the GML data should be sent to GML\n",
      "# for review before they are submitted for publication so we can\n",
      "# ensure that the quality and limitations of the data are accurately\n",
      "# represented.\n",
      "# \n",
      "# Contact:   Pieter Tans (303 497 6678; pieter.tans@noaa.gov)\n",
      "# \n",
      "# File Creation:  Fri Jan 21 05:00:12 2022\n",
      "# \n",
      "# RECIPROCITY\n",
      "# \n",
      "# Use of these data implies an agreement to reciprocate.\n",
      "# Laboratories making similar measurements agree to make their\n",
      "# own data available to the general public and to the scientific\n",
      "# community in an equally complete and easily accessible form.\n",
      "# Modelers are encouraged to make available to the community,\n",
      "# upon request, their own tools used in the interpretation\n",
      "# of the GML data, namely well documented model code, transport\n",
      "# fields, and additional information necessary for other\n",
      "# scientists to repeat the work and to run modified versions.\n",
      "# Model availability includes collaborative support for new\n",
      "# users of the models.\n",
      "# --------------------------------------------------------------------\n",
      "#  \n",
      "#  \n",
      "# See www.esrl.noaa.gov/gmd/ccgg/trends/ for additional details.\n",
      "#  \n",
      "# NOTE: DATA FOR THE LAST SEVERAL MONTHS ARE PRELIMINARY, ARE STILL SUBJECT\n",
      "# TO QUALITY CONTROL PROCEDURES.\n",
      "# NOTE: The week \"1 yr ago\" is exactly 365 days ago, and thus does not run from\n",
      "# Sunday through Saturday. 365 also ignores the possibility of a leap year.\n",
      "# The week \"10 yr ago\" is exactly 10*365 days +3 days (for leap years) ago.\n",
      "#  \n",
      "#      Start of week      CO2 molfrac           (-999.99 = no data)  increase\n",
      "# (yr, mon, day, decimal)    (ppm)  #days       1 yr ago  10 yr ago  since 1800\n",
      "  1974   5  19  1974.3795    333.37  5          -999.99   -999.99     50.40\n",
      "  1974   5  26  1974.3986    332.95  6          -999.99   -999.99     50.06\n",
      "  1974   6   2  1974.4178    332.35  5          -999.99   -999.99     49.60\n",
      "  1974   6   9  1974.4370    332.20  7          -999.99   -999.99     49.65\n",
      "  1974   6  16  1974.4562    332.37  7          -999.99   -999.99     50.06\n",
      "  1974   6  23  1974.4753    331.73  5          -999.99   -999.99     49.72\n",
      "  1974   6  30  1974.4945    331.68  6          -999.99   -999.99     50.02\n",
      "  1974   7   7  1974.5137    331.46  6          -999.99   -999.99     50.20\n",
      "  1974   7  14  1974.5329    330.83  5          -999.99   -999.99     50.01\n",
      "  1974   7  21  1974.5521    330.76  7          -999.99   -999.99     50.41\n",
      "  1974   7  28  1974.5712    329.80  4          -999.99   -999.99     49.97\n",
      "  1974   8   4  1974.5904    329.85  5          -999.99   -999.99     50.54\n"
     ]
    }
   ],
   "source": [
    "with open(\"co2_weekly_mlo.txt\", \"r\") as f:\n",
    "    for i, line in enumerate(f.readlines()):\n",
    "        if i > 60: # let's print only the first 60 lines only\n",
    "            break\n",
    "        print(line[:-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fce4d8e8",
   "metadata": {},
   "source": [
    "We start by loading the necessary Python modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "300cf8d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.gaussian_process import GaussianProcessRegressor\n",
    "from sklearn.gaussian_process.kernels import RBF, ExpSineSquared, WhiteKernel, RationalQuadratic"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ecd6a69",
   "metadata": {},
   "source": [
    "Let us now load the data downloaded using pandas. If you are not familiar with Pandas, take a look at the documentation for a quick introduction at https://pandas.pydata.org/docs/user_guide/10min.html\n",
    "\n",
    "We can easily parse the data file using Pandas' read_csv function. It would require some coding to do it manually. As Pandas does not detect the column names, we have to specify it manually in the names parameter below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4959a292",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"co2_weekly_mlo.txt\",\n",
    "                   delim_whitespace=True,\n",
    "                   comment='#',\n",
    "                   header=None,\n",
    "                   names=[\"year\", \"month\", \"day\", \"decimal\",\n",
    "                          \"co2\",\n",
    "                          \"ndays\",\n",
    "                          \"last_year\", \"last_decade\",\n",
    "                          \"increase\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8295e8a",
   "metadata": {},
   "source": [
    "Let's print out the dataset read first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "024fb65a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "      <th>decimal</th>\n",
       "      <th>co2</th>\n",
       "      <th>ndays</th>\n",
       "      <th>last_year</th>\n",
       "      <th>last_decade</th>\n",
       "      <th>increase</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1974</td>\n",
       "      <td>5</td>\n",
       "      <td>19</td>\n",
       "      <td>1974.3795</td>\n",
       "      <td>333.37</td>\n",
       "      <td>5</td>\n",
       "      <td>-999.99</td>\n",
       "      <td>-999.99</td>\n",
       "      <td>50.40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1974</td>\n",
       "      <td>5</td>\n",
       "      <td>26</td>\n",
       "      <td>1974.3986</td>\n",
       "      <td>332.95</td>\n",
       "      <td>6</td>\n",
       "      <td>-999.99</td>\n",
       "      <td>-999.99</td>\n",
       "      <td>50.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1974</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>1974.4178</td>\n",
       "      <td>332.35</td>\n",
       "      <td>5</td>\n",
       "      <td>-999.99</td>\n",
       "      <td>-999.99</td>\n",
       "      <td>49.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1974</td>\n",
       "      <td>6</td>\n",
       "      <td>9</td>\n",
       "      <td>1974.4370</td>\n",
       "      <td>332.20</td>\n",
       "      <td>7</td>\n",
       "      <td>-999.99</td>\n",
       "      <td>-999.99</td>\n",
       "      <td>49.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1974</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>1974.4562</td>\n",
       "      <td>332.37</td>\n",
       "      <td>7</td>\n",
       "      <td>-999.99</td>\n",
       "      <td>-999.99</td>\n",
       "      <td>50.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2482</th>\n",
       "      <td>2021</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>2021.9466</td>\n",
       "      <td>416.45</td>\n",
       "      <td>7</td>\n",
       "      <td>414.00</td>\n",
       "      <td>392.19</td>\n",
       "      <td>137.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2483</th>\n",
       "      <td>2021</td>\n",
       "      <td>12</td>\n",
       "      <td>19</td>\n",
       "      <td>2021.9658</td>\n",
       "      <td>417.49</td>\n",
       "      <td>6</td>\n",
       "      <td>414.86</td>\n",
       "      <td>392.54</td>\n",
       "      <td>137.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2484</th>\n",
       "      <td>2021</td>\n",
       "      <td>12</td>\n",
       "      <td>26</td>\n",
       "      <td>2021.9849</td>\n",
       "      <td>417.46</td>\n",
       "      <td>7</td>\n",
       "      <td>415.32</td>\n",
       "      <td>392.43</td>\n",
       "      <td>137.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2485</th>\n",
       "      <td>2022</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2022.0041</td>\n",
       "      <td>417.43</td>\n",
       "      <td>6</td>\n",
       "      <td>415.44</td>\n",
       "      <td>393.18</td>\n",
       "      <td>137.40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2486</th>\n",
       "      <td>2022</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>2022.0233</td>\n",
       "      <td>418.33</td>\n",
       "      <td>7</td>\n",
       "      <td>415.06</td>\n",
       "      <td>393.99</td>\n",
       "      <td>138.10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2487 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      year  month  day    decimal     co2  ndays  last_year  last_decade  \\\n",
       "0     1974      5   19  1974.3795  333.37      5    -999.99      -999.99   \n",
       "1     1974      5   26  1974.3986  332.95      6    -999.99      -999.99   \n",
       "2     1974      6    2  1974.4178  332.35      5    -999.99      -999.99   \n",
       "3     1974      6    9  1974.4370  332.20      7    -999.99      -999.99   \n",
       "4     1974      6   16  1974.4562  332.37      7    -999.99      -999.99   \n",
       "...    ...    ...  ...        ...     ...    ...        ...          ...   \n",
       "2482  2021     12   12  2021.9466  416.45      7     414.00       392.19   \n",
       "2483  2021     12   19  2021.9658  417.49      6     414.86       392.54   \n",
       "2484  2021     12   26  2021.9849  417.46      7     415.32       392.43   \n",
       "2485  2022      1    2  2022.0041  417.43      6     415.44       393.18   \n",
       "2486  2022      1    9  2022.0233  418.33      7     415.06       393.99   \n",
       "\n",
       "      increase  \n",
       "0        50.40  \n",
       "1        50.06  \n",
       "2        49.60  \n",
       "3        49.65  \n",
       "4        50.06  \n",
       "...        ...  \n",
       "2482    137.16  \n",
       "2483    137.93  \n",
       "2484    137.66  \n",
       "2485    137.40  \n",
       "2486    138.10  \n",
       "\n",
       "[2487 rows x 9 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c178424",
   "metadata": {},
   "source": [
    "We can plot this fairly easily using Matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e63b38c5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 39176 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(8, 8))\n",
    "data.plot(x=\"decimal\", y=\"co2\", legend=False, ax=ax)\n",
    "ax.set(xlabel=\"Time [years]\", ylabel=r\"CO$_2$ [ppm]\", title=\"Atmospheric CO$_2$ concentration in Mauna Loa\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efd249fe",
   "metadata": {},
   "source": [
    "The spikes are related to missing data. We can clean those up and plot again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0837b3ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data[data.co2 > -999]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "333581b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 68156 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(8, 8))\n",
    "data.plot(x=\"decimal\", y=\"co2\", legend=False, ax=ax)\n",
    "ax.set(xlabel=\"Time [years]\", ylabel=r\"CO$_2$ [ppm]\", title=\"Atmospheric CO$_2$ concentration in Mauna Loa\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c62fd7f3",
   "metadata": {},
   "source": [
    "That's better.\n",
    "\n",
    "We can now try to model this behaviour with Gaussian Processes. The key parameter we must choose is the covariance matrix analytic form, which establishes how correlated two points in the time axis are.\n",
    "\n",
    "One simple assumption would be that two values of the CO2 concentration are highly correlated they happened at close by times. We use a Gaussian-based model (named here \"Radial Basis Function\", as this does not refer to a probability distribution) to establish that if two points are one year apart they are likely to be highly correlated.\n",
    "\n",
    "Additionally, we assume that there is a correlation between points within certain period, since we know that data from the same season in an year is expected to be correlated to data if that same season in the year (ie: CO$_2$ emissions have a rough periodicity of one year due to the season changes).\n",
    "\n",
    "Another term of the covariance matrix must be related to the random noise appearing in the data.\n",
    "\n",
    "All those assumptions can be motivated by an analysis of the covariance matrix of the data. That is, we can take the covariance matrix of all the data points and analyse it to establish whether our assumptions here are valid or not. A very long and detailed explanation on this point can be found in Bishop (2006), which has a very pedagogical chapter on Gaussian Processes. Online resources can be found on: http://www.gaussianprocess.org/\n",
    "\n",
    "Note that all of those are very strong assumptions, but they are made directly on the covariance matrix of the fit function. The actual numbers used below are taken from an example, but they will be optimized in the fit below (unless explicitly asked not to).\n",
    "\n",
    "Note: this notebook is a simplified example with extra explanations, with lots of material taken from https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_co2.html -- take a look at the original example for more details.\n",
    "\n",
    "We also assume the data within an year from each other is highly correlated and should show a periodicity of 1 year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a31d7eca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Kernel with parameters given in GPML book\n",
    "\n",
    "# these hyper parameters are optimised in the GP fit\n",
    "# we provide here only a starting value\n",
    "\n",
    "# Kernel with optimized initial parameters\n",
    "k1 = 50.0 ** 2 * RBF(length_scale=50.0)  # long term smooth rising trend\n",
    "k2 = (\n",
    "    2.0 ** 2\n",
    "    * RBF(length_scale=100.0)\n",
    "    * ExpSineSquared(length_scale=1.0, periodicity=1.0, periodicity_bounds=\"fixed\")\n",
    ")  # seasonal component\n",
    "# medium term irregularities\n",
    "k3 = 0.5 ** 2 * RationalQuadratic(length_scale=1.0, alpha=1.0)\n",
    "k4 = 0.1 ** 2 * RBF(length_scale=0.1) + WhiteKernel(\n",
    "    noise_level=0.1 ** 2, noise_level_bounds=(1e-5, np.inf)\n",
    ")  # noise terms\n",
    "kernel = k1 + k2 + k3 + k4\n",
    "\n",
    "# simpler kernel: RBF kernel with white noise on likelihood\n",
    "# what happens if you use this instead?\n",
    "#kernel = (10.0 ** 2 * RBF(length_scale=10.0)\n",
    "#          + WhiteKernel(noise_level=1**2, noise_level_bounds=(1e-5, np.inf))\n",
    "#         )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ad8ddf0",
   "metadata": {},
   "source": [
    "The kernel hyper-parameters are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "032ada0d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50**2 * RBF(length_scale=50) + 2**2 * RBF(length_scale=100) * ExpSineSquared(length_scale=1, periodicity=1) + 0.5**2 * RationalQuadratic(alpha=1, length_scale=1) + 0.1**2 * RBF(length_scale=0.1) + WhiteKernel(noise_level=0.01)\n"
     ]
    }
   ],
   "source": [
    "print(kernel)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1956bd05-4658-4c41-931c-8cb003534eb4",
   "metadata": {},
   "source": [
    "We can now fit the data that we have. But let us not fit the full data: we can fit data using only information until 2014 to use the period between 2014 and now as an estimate of the fit quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "22046fb8-135c-4048-a0c5-ceb1e8190049",
   "metadata": {},
   "outputs": [],
   "source": [
    "fit_data = data.loc[data.year <= 2014, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4412a9a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "gp = GaussianProcessRegressor(kernel=kernel, normalize_y=True)\n",
    "time = fit_data.decimal.to_numpy()[:, np.newaxis]\n",
    "co2 = fit_data.co2.to_numpy()[:, np.newaxis]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "88bcd2c1-c9ae-4f89-ba9a-96d731907138",
   "metadata": {},
   "outputs": [],
   "source": [
    "gp = gp.fit(time, co2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb69bb80",
   "metadata": {},
   "source": [
    "These are the hyper-parameters after the fit and the log-likelihood given the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "667db862",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPML kernel: 3.27**2 * RBF(length_scale=37.8) + 0.175**2 * RBF(length_scale=290) * ExpSineSquared(length_scale=1.59, periodicity=1) + 0.322**2 * RationalQuadratic(alpha=0.00131, length_scale=6.52) + 0.0121**2 * RBF(length_scale=0.02) + WhiteKernel(noise_level=0.000209)\n",
      "Log-marginal-likelihood: 5275.122\n"
     ]
    }
   ],
   "source": [
    "print(\"GPML kernel: %s\" % gp.kernel_)\n",
    "print(\"Log-marginal-likelihood: %.3f\" % gp.log_marginal_likelihood(gp.kernel_.theta))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4968439e",
   "metadata": {},
   "source": [
    "Now we can use the fit model to predict how the CO2 concentration will evolve at other times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ef6b7762",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_time = np.linspace(time.min(), time.max()+20, 1000)[:, np.newaxis]\n",
    "co2_pred, co2_std = gp.predict(new_time, return_std=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c471fa1-8221-4d0a-bcb8-bc2515487c7b",
   "metadata": {},
   "source": [
    "First let's plot the prediction with uncertainty only:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "262c8e85-6f9c-447f-9e50-a356267e0597",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 74832 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(8, 8))\n",
    "ax.plot(new_time[:,0],\n",
    "        co2_pred,\n",
    "        color=\"b\",\n",
    "        lw=2,\n",
    "        label=\"Mean expectation\")\n",
    "ax.fill_between(new_time[:,0],\n",
    "                co2_pred - co2_std,\n",
    "                co2_pred + co2_std,\n",
    "                color=\"b\",\n",
    "                alpha=0.4,\n",
    "                label=\"68% Confidence Level uncertainty\")\n",
    "ax.set(xlabel=\"Time [years]\", ylabel=r\"CO$_2$ [ppm]\", title=\"Atmospheric CO$_2$ concentration in Mauna Loa\")\n",
    "ax.legend(frameon=False, loc=\"lower right\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3397756c-ea3a-4cf8-a4cc-643837baca33",
   "metadata": {},
   "source": [
    "Now let's plot the actual dat above it, to check how far off we are. Let's also plot the training data and full data separately, so we can see where data not used in the fit starts: this is where our uncertainty should start becoming bigger if everything worked out correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5b263358",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 98244 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(8, 8))\n",
    "ax.scatter(time, co2, color='red', marker='o', label=\"Fit data\")\n",
    "ax.scatter(data.decimal, data.co2, color='green', marker='.', label=\"All data\")\n",
    "ax.fill_between(new_time[:,0],\n",
    "                co2_pred - co2_std,\n",
    "                co2_pred + co2_std,\n",
    "                color=\"b\",\n",
    "                alpha=0.4,\n",
    "                label=\"Predictions\")\n",
    "ax.set(xlabel=\"Time [years]\", ylabel=r\"CO$_2$ [ppm]\", title=\"Atmospheric CO$_2$ concentration in Mauna Loa\")\n",
    "ax.legend(frameon=False, loc=\"lower right\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4beb5578-8437-4696-994a-7cde29266f15",
   "metadata": {},
   "source": [
    "Try changing the kernel above and check what happens. Notice how large an effect the kernel definition has in the results!\n",
    "\n",
    "The kernel is one of our key assumptions here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b625b695",
   "metadata": {},
   "source": [
    "Many more details on the kernel selection as well as further details in this example can be seen in the following references:\n",
    "\n",
    "https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_co2.html\n",
    "\n",
    "https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_noisy_targets.html#sphx-glr-auto-examples-gaussian-process-plot-gpr-noisy-targets-py\n",
    "\n",
    "To read more on Gaussian Processes:\n",
    "\n",
    "http://www.gaussianprocess.org/gpml/chapters/RW.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "663ff599-7b8d-4f13-a02f-3b9a7808814e",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
