{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "73b6c8c9",
   "metadata": {},
   "source": [
    "# Non-Linear Modeling\n",
    "\n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/intro-stat-learning/ISLP_labs/blob/v2.2/Ch07-nonlin-lab.ipynb\">\n",
    "<img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "\n",
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/intro-stat-learning/ISLP_labs/v2.2?labpath=Ch07-nonlin-lab.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c645659",
   "metadata": {},
   "source": [
    "In this lab, we demonstrate some of the nonlinear models discussed in\n",
    "this chapter. We use the `Wage`  data as a running example, and show that many of the complex non-linear fitting procedures discussed can easily be implemented in `Python`.\n",
    "\n",
    "As usual, we start with some of our standard imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4f8339f1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:34.911110Z",
     "iopub.status.busy": "2024-06-04T23:19:34.910732Z",
     "iopub.status.idle": "2024-06-04T23:19:35.700317Z",
     "shell.execute_reply": "2024-06-04T23:19:35.700028Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np, pandas as pd\n",
    "from matplotlib.pyplot import subplots\n",
    "import statsmodels.api as sm\n",
    "from ISLP import load_data\n",
    "from ISLP.models import (summarize,\n",
    "                         poly,\n",
    "                         ModelSpec as MS)\n",
    "from statsmodels.stats.anova import anova_lm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8fa3d7b",
   "metadata": {},
   "source": [
    "We again collect the new imports\n",
    "needed for this lab. Many of these are developed specifically for the\n",
    "`ISLP` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "816d4b81",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:35.702094Z",
     "iopub.status.busy": "2024-06-04T23:19:35.701965Z",
     "iopub.status.idle": "2024-06-04T23:19:35.722229Z",
     "shell.execute_reply": "2024-06-04T23:19:35.722033Z"
    }
   },
   "outputs": [],
   "source": [
    "from pygam import (s as s_gam,\n",
    "                   l as l_gam,\n",
    "                   f as f_gam,\n",
    "                   LinearGAM,\n",
    "                   LogisticGAM)\n",
    "\n",
    "from ISLP.transforms import (BSpline,\n",
    "                             NaturalSpline)\n",
    "from ISLP.models import bs, ns\n",
    "from ISLP.pygam import (approx_lam,\n",
    "                        degrees_of_freedom,\n",
    "                        plot as plot_gam,\n",
    "                        anova as anova_gam)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da7fd8b8",
   "metadata": {},
   "source": [
    "## Polynomial Regression and Step Functions\n",
    "We start by demonstrating how Figure~\\ref{Ch7:fig:poly} can be reproduced.\n",
    "Let's  begin by loading the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "bd30b197",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:35.723538Z",
     "iopub.status.busy": "2024-06-04T23:19:35.723455Z",
     "iopub.status.idle": "2024-06-04T23:19:35.730561Z",
     "shell.execute_reply": "2024-06-04T23:19:35.730367Z"
    }
   },
   "outputs": [],
   "source": [
    "Wage = load_data('Wage')\n",
    "y = Wage['wage']\n",
    "age = Wage['age']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2539b250",
   "metadata": {},
   "source": [
    "Throughout most of this lab, our response is `Wage['wage']`, which\n",
    "we have stored as `y` above. \n",
    "As in Section~\\ref{Ch3-linreg-lab:non-linear-transformations-of-the-predictors}, we will use the `poly()` function to create a model matrix\n",
    "that will fit a $4$th degree polynomial in `age`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fe40e0fb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:35.731830Z",
     "iopub.status.busy": "2024-06-04T23:19:35.731745Z",
     "iopub.status.idle": "2024-06-04T23:19:35.758719Z",
     "shell.execute_reply": "2024-06-04T23:19:35.758161Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>111.7036</td>\n",
       "      <td>0.729</td>\n",
       "      <td>153.283</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(age, degree=4)[0]</th>\n",
       "      <td>447.0679</td>\n",
       "      <td>39.915</td>\n",
       "      <td>11.201</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(age, degree=4)[1]</th>\n",
       "      <td>-478.3158</td>\n",
       "      <td>39.915</td>\n",
       "      <td>-11.983</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(age, degree=4)[2]</th>\n",
       "      <td>125.5217</td>\n",
       "      <td>39.915</td>\n",
       "      <td>3.145</td>\n",
       "      <td>0.002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(age, degree=4)[3]</th>\n",
       "      <td>-77.9112</td>\n",
       "      <td>39.915</td>\n",
       "      <td>-1.952</td>\n",
       "      <td>0.051</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            coef  std err        t  P>|t|\n",
       "intercept               111.7036    0.729  153.283  0.000\n",
       "poly(age, degree=4)[0]  447.0679   39.915   11.201  0.000\n",
       "poly(age, degree=4)[1] -478.3158   39.915  -11.983  0.000\n",
       "poly(age, degree=4)[2]  125.5217   39.915    3.145  0.002\n",
       "poly(age, degree=4)[3]  -77.9112   39.915   -1.952  0.051"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poly_age = MS([poly('age', degree=4)]).fit(Wage)\n",
    "M = sm.OLS(y, poly_age.transform(Wage)).fit()\n",
    "summarize(M)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be02ce11",
   "metadata": {},
   "source": [
    "This polynomial is constructed using the function `poly()`,\n",
    "which creates\n",
    "a special *transformer* `Poly()` (using `sklearn` terminology\n",
    "for feature transformations such as `PCA()` seen in Section \\ref{Ch6-varselect-lab:principal-components-regression}) which\n",
    "allows for easy evaluation of the polynomial at new data points. Here `poly()` is referred to as a *helper* function, and sets up the transformation; `Poly()` is the actual workhorse that computes the transformation. See also \n",
    "the \n",
    "discussion of transformations on\n",
    "page~\\pageref{Ch3-linreg-lab:using-transformations-fit-and-transform}. \n",
    "\n",
    "In the code above, the first line executes the `fit()` method\n",
    "using the dataframe\n",
    "`Wage`. This recomputes and stores as attributes any parameters needed by `Poly()`\n",
    "on the training data, and these will be used on all subsequent\n",
    "evaluations of the `transform()` method. For example, it is used\n",
    "on the second line, as well as in the plotting function developed below.\n",
    "\n",
    "\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64316183",
   "metadata": {},
   "source": [
    "We now create a grid of values for `age` at which we want\n",
    "predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "23a9ff8e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:35.762173Z",
     "iopub.status.busy": "2024-06-04T23:19:35.761857Z",
     "iopub.status.idle": "2024-06-04T23:19:35.766441Z",
     "shell.execute_reply": "2024-06-04T23:19:35.765894Z"
    }
   },
   "outputs": [],
   "source": [
    "age_grid = np.linspace(age.min(),\n",
    "                       age.max(),\n",
    "                       100)\n",
    "age_df = pd.DataFrame({'age': age_grid})\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c10f12d4",
   "metadata": {},
   "source": [
    "Finally, we wish to plot the data and add the fit from the fourth-degree polynomial. As we will make\n",
    "several similar plots below, we first write a function\n",
    "to create all the ingredients and produce the plot.  Our function\n",
    "takes in a model specification (here a basis specified by a\n",
    "transform), as well as a grid of `age` values. The function\n",
    "produces a fitted curve as well as 95% confidence bands. By using\n",
    "an argument for `basis` we can produce and plot the results with several different\n",
    "transforms, such as the splines we will see shortly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ca7a1d45",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:35.771011Z",
     "iopub.status.busy": "2024-06-04T23:19:35.770581Z",
     "iopub.status.idle": "2024-06-04T23:19:35.776229Z",
     "shell.execute_reply": "2024-06-04T23:19:35.775762Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "def plot_wage_fit(age_df, \n",
    "                  basis,\n",
    "                  title):\n",
    "\n",
    "    X = basis.transform(Wage)\n",
    "    Xnew = basis.transform(age_df)\n",
    "    M = sm.OLS(y, X).fit()\n",
    "    preds = M.get_prediction(Xnew)\n",
    "    bands = preds.conf_int(alpha=0.05)\n",
    "    fig, ax = subplots(figsize=(8,8))\n",
    "    ax.scatter(age,\n",
    "               y,\n",
    "               facecolor='gray',\n",
    "               alpha=0.5)\n",
    "    for val, ls in zip([preds.predicted_mean,\n",
    "                      bands[:,0],\n",
    "                      bands[:,1]],\n",
    "                     ['b','r--','r--']):\n",
    "        ax.plot(age_df.values, val, ls, linewidth=3)\n",
    "    ax.set_title(title, fontsize=20)\n",
    "    ax.set_xlabel('Age', fontsize=20)\n",
    "    ax.set_ylabel('Wage', fontsize=20);\n",
    "    return ax\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14d6a540",
   "metadata": {},
   "source": [
    "We include an argument `alpha` to `ax.scatter()`\n",
    "to add some transparency to the points. This provides a visual indication\n",
    "of density. Notice the use of the `zip()` function in the\n",
    "`for` loop above (see Section~\\ref{Ch2-statlearn-lab:for-loops}).\n",
    "We have three lines to plot, each with different colors and line\n",
    "types. Here `zip()` conveniently bundles these together as\n",
    "iterators in the loop. {In `Python`{} speak, an \"iterator\" is an object with a finite number of values, that can be iterated on, as in a loop.}\n",
    "\n",
    "We now plot the fit of the fourth-degree polynomial using this\n",
    "function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b05a60ba",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:35.779177Z",
     "iopub.status.busy": "2024-06-04T23:19:35.778961Z",
     "iopub.status.idle": "2024-06-04T23:19:36.009418Z",
     "shell.execute_reply": "2024-06-04T23:19:36.008910Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 180448 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_wage_fit(age_df, \n",
    "              poly_age,\n",
    "              'Degree-4 Polynomial');\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "648104d9",
   "metadata": {},
   "source": [
    " \n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eed651b8",
   "metadata": {},
   "source": [
    "With  polynomial regression we must decide on the degree of\n",
    "the polynomial to use. Sometimes we just wing it, and decide to use\n",
    "second or third degree polynomials, simply to obtain a nonlinear fit. But we can\n",
    "make such a decision in a more systematic way. One way to do this is through hypothesis\n",
    "tests, which we demonstrate here. We now fit a series of models ranging from\n",
    "linear (degree-one) to degree-five polynomials,\n",
    "and look to determine the simplest model that is sufficient to\n",
    "explain the relationship between `wage` and `age`. We use the\n",
    "`anova_lm()`  function, which performs a series of ANOVA\n",
    "tests.\n",
    "An \\emph{analysis of\n",
    "  variance}  or ANOVA tests the null\n",
    "hypothesis that a model $\\mathcal{M}_1$ is sufficient to explain the\n",
    "data against the alternative hypothesis that a more complex model\n",
    "$\\mathcal{M}_2$ is required. The determination is based on an F-test.\n",
    "To perform the test, the models  $\\mathcal{M}_1$ and $\\mathcal{M}_2$ must be *nested*:\n",
    "the space spanned by the predictors in $\\mathcal{M}_1$ must be a subspace of the\n",
    "space spanned by the predictors in $\\mathcal{M}_2$. In this case, we\n",
    "fit five different polynomial\n",
    "models and sequentially compare the simpler model to the more complex\n",
    "model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c338a92a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.012673Z",
     "iopub.status.busy": "2024-06-04T23:19:36.012313Z",
     "iopub.status.idle": "2024-06-04T23:19:36.072501Z",
     "shell.execute_reply": "2024-06-04T23:19:36.071990Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>df_resid</th>\n",
       "      <th>ssr</th>\n",
       "      <th>df_diff</th>\n",
       "      <th>ss_diff</th>\n",
       "      <th>F</th>\n",
       "      <th>Pr(&gt;F)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2998.0</td>\n",
       "      <td>5.022216e+06</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2997.0</td>\n",
       "      <td>4.793430e+06</td>\n",
       "      <td>1.0</td>\n",
       "      <td>228786.010128</td>\n",
       "      <td>143.593107</td>\n",
       "      <td>2.363850e-32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2996.0</td>\n",
       "      <td>4.777674e+06</td>\n",
       "      <td>1.0</td>\n",
       "      <td>15755.693664</td>\n",
       "      <td>9.888756</td>\n",
       "      <td>1.679202e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2995.0</td>\n",
       "      <td>4.771604e+06</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6070.152124</td>\n",
       "      <td>3.809813</td>\n",
       "      <td>5.104620e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2994.0</td>\n",
       "      <td>4.770322e+06</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1282.563017</td>\n",
       "      <td>0.804976</td>\n",
       "      <td>3.696820e-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   df_resid           ssr  df_diff        ss_diff           F        Pr(>F)\n",
       "0    2998.0  5.022216e+06      0.0            NaN         NaN           NaN\n",
       "1    2997.0  4.793430e+06      1.0  228786.010128  143.593107  2.363850e-32\n",
       "2    2996.0  4.777674e+06      1.0   15755.693664    9.888756  1.679202e-03\n",
       "3    2995.0  4.771604e+06      1.0    6070.152124    3.809813  5.104620e-02\n",
       "4    2994.0  4.770322e+06      1.0    1282.563017    0.804976  3.696820e-01"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "models = [MS([poly('age', degree=d)]) \n",
    "          for d in range(1, 6)]\n",
    "Xs = [model.fit_transform(Wage) for model in models]\n",
    "anova_lm(*[sm.OLS(y, X_).fit()\n",
    "           for X_ in Xs])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a8c2c29",
   "metadata": {},
   "source": [
    "Notice the `*` in the `anova_lm()` line above. This\n",
    "function takes a variable number of non-keyword arguments, in this case fitted models.\n",
    "When these models are provided as a list (as is done here), it must be \n",
    "prefixed by `*`.\n",
    "\n",
    "The p-value comparing the linear `models[0]` to the quadratic\n",
    "`models[1]` is essentially zero, indicating that a linear\n",
    "fit is not sufficient. {Indexing starting at zero is confusing for the polynomial degree example, since `models[1]` is quadratic rather than linear!} Similarly the p-value comparing the quadratic\n",
    "`models[1]` to the cubic `models[2]` is very low (0.0017), so the\n",
    "quadratic fit is also insufficient. The p-value comparing the cubic\n",
    "and degree-four polynomials, `models[2]` and `models[3]`, is\n",
    "approximately 5%, while the degree-five polynomial `models[4]` seems\n",
    "unnecessary because its p-value is 0.37. Hence, either a cubic or a\n",
    "quartic polynomial appear to provide a reasonable fit to the data, but\n",
    "lower- or higher-order models are not justified.\n",
    "\n",
    "In this case, instead of using the `anova()`  function, we could\n",
    "have obtained these p-values more succinctly by exploiting the fact\n",
    "that `poly()`  creates orthogonal polynomials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a1f74b79",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.075597Z",
     "iopub.status.busy": "2024-06-04T23:19:36.075369Z",
     "iopub.status.idle": "2024-06-04T23:19:36.090416Z",
     "shell.execute_reply": "2024-06-04T23:19:36.089945Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>111.7036</td>\n",
       "      <td>0.729</td>\n",
       "      <td>153.283</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(age, degree=4)[0]</th>\n",
       "      <td>447.0679</td>\n",
       "      <td>39.915</td>\n",
       "      <td>11.201</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(age, degree=4)[1]</th>\n",
       "      <td>-478.3158</td>\n",
       "      <td>39.915</td>\n",
       "      <td>-11.983</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(age, degree=4)[2]</th>\n",
       "      <td>125.5217</td>\n",
       "      <td>39.915</td>\n",
       "      <td>3.145</td>\n",
       "      <td>0.002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(age, degree=4)[3]</th>\n",
       "      <td>-77.9112</td>\n",
       "      <td>39.915</td>\n",
       "      <td>-1.952</td>\n",
       "      <td>0.051</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            coef  std err        t  P>|t|\n",
       "intercept               111.7036    0.729  153.283  0.000\n",
       "poly(age, degree=4)[0]  447.0679   39.915   11.201  0.000\n",
       "poly(age, degree=4)[1] -478.3158   39.915  -11.983  0.000\n",
       "poly(age, degree=4)[2]  125.5217   39.915    3.145  0.002\n",
       "poly(age, degree=4)[3]  -77.9112   39.915   -1.952  0.051"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summarize(M)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea22f1d6",
   "metadata": {},
   "source": [
    "Notice that the p-values are the same, and in fact the square of\n",
    "the  t-statistics are equal to the F-statistics from the\n",
    "`anova_lm()`  function;  for example: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "570892d1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.092926Z",
     "iopub.status.busy": "2024-06-04T23:19:36.092726Z",
     "iopub.status.idle": "2024-06-04T23:19:36.096105Z",
     "shell.execute_reply": "2024-06-04T23:19:36.095602Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "143.59228900000002"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(-11.983)**2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcb5ee4d",
   "metadata": {},
   "source": [
    "However, the ANOVA method works whether or not we used orthogonal\n",
    "polynomials, provided the models are nested. For example, we can use\n",
    "`anova_lm()`  to compare the following three\n",
    "models, which all have a linear term in `education` and a\n",
    "polynomial in `age` of different degrees:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3ff20916",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.098614Z",
     "iopub.status.busy": "2024-06-04T23:19:36.098427Z",
     "iopub.status.idle": "2024-06-04T23:19:36.147734Z",
     "shell.execute_reply": "2024-06-04T23:19:36.145914Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>df_resid</th>\n",
       "      <th>ssr</th>\n",
       "      <th>df_diff</th>\n",
       "      <th>ss_diff</th>\n",
       "      <th>F</th>\n",
       "      <th>Pr(&gt;F)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2997.0</td>\n",
       "      <td>3.902335e+06</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2996.0</td>\n",
       "      <td>3.759472e+06</td>\n",
       "      <td>1.0</td>\n",
       "      <td>142862.701185</td>\n",
       "      <td>113.991883</td>\n",
       "      <td>3.838075e-26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2995.0</td>\n",
       "      <td>3.753546e+06</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5926.207070</td>\n",
       "      <td>4.728593</td>\n",
       "      <td>2.974318e-02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   df_resid           ssr  df_diff        ss_diff           F        Pr(>F)\n",
       "0    2997.0  3.902335e+06      0.0            NaN         NaN           NaN\n",
       "1    2996.0  3.759472e+06      1.0  142862.701185  113.991883  3.838075e-26\n",
       "2    2995.0  3.753546e+06      1.0    5926.207070    4.728593  2.974318e-02"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "models = [MS(['education', poly('age', degree=d)])\n",
    "          for d in range(1, 4)]\n",
    "XEs = [model.fit_transform(Wage)\n",
    "       for model in models]\n",
    "anova_lm(*[sm.OLS(y, X_).fit() for X_ in XEs])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12da652b",
   "metadata": {},
   "source": [
    "As an alternative to using hypothesis tests and ANOVA, we could choose\n",
    "the polynomial degree using cross-validation, as discussed in Chapter~\\ref{Ch5:resample}.\n",
    "\n",
    "Next we consider the task of predicting whether an individual earns\n",
    "more than $250,000 per year. We proceed much as before, except\n",
    "that first we create the appropriate response vector, and then apply\n",
    "the `glm()`  function using the binomial family in order\n",
    "to fit a polynomial logistic regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2952bde3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.154502Z",
     "iopub.status.busy": "2024-06-04T23:19:36.154251Z",
     "iopub.status.idle": "2024-06-04T23:19:36.179360Z",
     "shell.execute_reply": "2024-06-04T23:19:36.177817Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>z</th>\n",
       "      <th>P&gt;|z|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>-4.3012</td>\n",
       "      <td>0.345</td>\n",
       "      <td>-12.457</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(age, degree=4)[0]</th>\n",
       "      <td>71.9642</td>\n",
       "      <td>26.133</td>\n",
       "      <td>2.754</td>\n",
       "      <td>0.006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(age, degree=4)[1]</th>\n",
       "      <td>-85.7729</td>\n",
       "      <td>35.929</td>\n",
       "      <td>-2.387</td>\n",
       "      <td>0.017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(age, degree=4)[2]</th>\n",
       "      <td>34.1626</td>\n",
       "      <td>19.697</td>\n",
       "      <td>1.734</td>\n",
       "      <td>0.083</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(age, degree=4)[3]</th>\n",
       "      <td>-47.4008</td>\n",
       "      <td>24.105</td>\n",
       "      <td>-1.966</td>\n",
       "      <td>0.049</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           coef  std err       z  P>|z|\n",
       "intercept               -4.3012    0.345 -12.457  0.000\n",
       "poly(age, degree=4)[0]  71.9642   26.133   2.754  0.006\n",
       "poly(age, degree=4)[1] -85.7729   35.929  -2.387  0.017\n",
       "poly(age, degree=4)[2]  34.1626   19.697   1.734  0.083\n",
       "poly(age, degree=4)[3] -47.4008   24.105  -1.966  0.049"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = poly_age.transform(Wage)\n",
    "high_earn = Wage['high_earn'] = y > 250 # shorthand\n",
    "glm = sm.GLM(y > 250,\n",
    "             X,\n",
    "             family=sm.families.Binomial())\n",
    "B = glm.fit()\n",
    "summarize(B)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "907c249a",
   "metadata": {},
   "source": [
    "Once again, we make predictions using the `get_prediction()`  method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c999b609",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.182205Z",
     "iopub.status.busy": "2024-06-04T23:19:36.181928Z",
     "iopub.status.idle": "2024-06-04T23:19:36.188474Z",
     "shell.execute_reply": "2024-06-04T23:19:36.187832Z"
    }
   },
   "outputs": [],
   "source": [
    "newX = poly_age.transform(age_df)\n",
    "preds = B.get_prediction(newX)\n",
    "bands = preds.conf_int(alpha=0.05)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37cf225e",
   "metadata": {},
   "source": [
    "We now plot the estimated relationship."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5f18a110",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.191550Z",
     "iopub.status.busy": "2024-06-04T23:19:36.191370Z",
     "iopub.status.idle": "2024-06-04T23:19:36.311190Z",
     "shell.execute_reply": "2024-06-04T23:19:36.310416Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 64956 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = subplots(figsize=(8,8))\n",
    "rng = np.random.default_rng(0)\n",
    "ax.scatter(age +\n",
    "           0.2 * rng.uniform(size=y.shape[0]),\n",
    "           np.where(high_earn, 0.198, 0.002),\n",
    "           fc='gray',\n",
    "           marker='|')\n",
    "for val, ls in zip([preds.predicted_mean,\n",
    "                  bands[:,0],\n",
    "                  bands[:,1]],\n",
    "                 ['b','r--','r--']):\n",
    "    ax.plot(age_df.values, val, ls, linewidth=3)\n",
    "ax.set_title('Degree-4 Polynomial', fontsize=20)\n",
    "ax.set_xlabel('Age', fontsize=20)\n",
    "ax.set_ylim([0,0.2])\n",
    "ax.set_ylabel('P(Wage > 250)', fontsize=20);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ad21a81",
   "metadata": {},
   "source": [
    "We have drawn the `age` values corresponding to the observations with\n",
    "`wage` values above 250 as gray marks on the top of the plot, and\n",
    "those with `wage` values below 250 are shown as gray marks on the\n",
    "bottom of the plot. We added a small amount of noise to jitter\n",
    "the `age` values a bit so that observations with the same `age`\n",
    "value do not cover each other up. This type of plot is often called a\n",
    "*rug plot*.\n",
    "\n",
    "In order to fit a step function, as discussed in\n",
    "Section~\\ref{Ch7:sec:scolstep-function},   we first use the `pd.qcut()`\n",
    "function to discretize `age` based on quantiles.  Then  we use `pd.get_dummies()` to create the\n",
    "columns of the model matrix for this categorical variable. Note that this function will\n",
    "include *all* columns for a given categorical, rather than the usual approach which drops one\n",
    "of the levels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "62e053ea",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.315065Z",
     "iopub.status.busy": "2024-06-04T23:19:36.314672Z",
     "iopub.status.idle": "2024-06-04T23:19:36.333569Z",
     "shell.execute_reply": "2024-06-04T23:19:36.333006Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>(17.999, 33.75]</th>\n",
       "      <td>94.1584</td>\n",
       "      <td>1.478</td>\n",
       "      <td>63.692</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(33.75, 42.0]</th>\n",
       "      <td>116.6608</td>\n",
       "      <td>1.470</td>\n",
       "      <td>79.385</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(42.0, 51.0]</th>\n",
       "      <td>119.1887</td>\n",
       "      <td>1.416</td>\n",
       "      <td>84.147</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(51.0, 80.0]</th>\n",
       "      <td>116.5717</td>\n",
       "      <td>1.559</td>\n",
       "      <td>74.751</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     coef  std err       t  P>|t|\n",
       "(17.999, 33.75]   94.1584    1.478  63.692    0.0\n",
       "(33.75, 42.0]    116.6608    1.470  79.385    0.0\n",
       "(42.0, 51.0]     119.1887    1.416  84.147    0.0\n",
       "(51.0, 80.0]     116.5717    1.559  74.751    0.0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cut_age = pd.qcut(age, 4)\n",
    "summarize(sm.OLS(y, pd.get_dummies(cut_age)).fit())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbe7cfaa",
   "metadata": {},
   "source": [
    "Here `pd.qcut()`  automatically picked the cutpoints based on the quantiles 25%, 50% and 75%, which results in four regions.  We could also have specified our own\n",
    "quantiles directly instead of the argument `4`. For cuts not based\n",
    "on quantiles we would use the `pd.cut()` function.\n",
    "The function `pd.qcut()` (and `pd.cut()`) returns an ordered categorical variable.\n",
    "  The regression model then creates a set of\n",
    "dummy variables for use in the regression. Since `age` is the only variable in the model, the value $94,158.40 is the average salary for those under 33.75 years of\n",
    "age, and the other coefficients are the average\n",
    "salary for those in the other age groups.  We can produce\n",
    "predictions and plots just as we did in the case of the polynomial\n",
    "fit."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81d470c1",
   "metadata": {},
   "source": [
    "## Splines\n",
    "In order to fit regression splines, we use transforms\n",
    "from the `ISLP` package. The actual spline\n",
    "evaluation functions are in the `scipy.interpolate` package;\n",
    "we have simply wrapped them as transforms\n",
    "similar to `Poly()` and `PCA()`.\n",
    "\n",
    "In Section~\\ref{Ch7:sec:scolr-splin}, we saw\n",
    "that regression splines can be fit by constructing an appropriate\n",
    "matrix of basis functions.  The `BSpline()`  function generates the\n",
    "entire matrix of basis functions for splines with the specified set of\n",
    "knots. By default, the B-splines produced are cubic. To change the degree, use\n",
    "the argument `degree`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f032f577",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.336489Z",
     "iopub.status.busy": "2024-06-04T23:19:36.336258Z",
     "iopub.status.idle": "2024-06-04T23:19:36.343862Z",
     "shell.execute_reply": "2024-06-04T23:19:36.343366Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3000, 7)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bs_ = BSpline(internal_knots=[25,40,60], intercept=True).fit(age)\n",
    "bs_age = bs_.transform(age)\n",
    "bs_age.shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed2f4578",
   "metadata": {},
   "source": [
    "This results in a seven-column matrix, which is what is expected for a cubic-spline basis with 3 interior knots. \n",
    "We can form this same matrix using the `bs()` object,\n",
    "which facilitates adding this to a model-matrix builder (as in `poly()` versus its workhorse `Poly()`) described in Section~\\ref{Ch7-nonlin-lab:polynomial-regression-and-step-functions}.\n",
    "\n",
    "We now fit a cubic spline model to the `Wage`  data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c3c38823",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.346759Z",
     "iopub.status.busy": "2024-06-04T23:19:36.346545Z",
     "iopub.status.idle": "2024-06-04T23:19:36.380669Z",
     "shell.execute_reply": "2024-06-04T23:19:36.380119Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>60.4937</td>\n",
       "      <td>9.460</td>\n",
       "      <td>6.394</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age, internal_knots=[25, 40, 60])[0]</th>\n",
       "      <td>3.9805</td>\n",
       "      <td>12.538</td>\n",
       "      <td>0.317</td>\n",
       "      <td>0.751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age, internal_knots=[25, 40, 60])[1]</th>\n",
       "      <td>44.6310</td>\n",
       "      <td>9.626</td>\n",
       "      <td>4.636</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age, internal_knots=[25, 40, 60])[2]</th>\n",
       "      <td>62.8388</td>\n",
       "      <td>10.755</td>\n",
       "      <td>5.843</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age, internal_knots=[25, 40, 60])[3]</th>\n",
       "      <td>55.9908</td>\n",
       "      <td>10.706</td>\n",
       "      <td>5.230</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age, internal_knots=[25, 40, 60])[4]</th>\n",
       "      <td>50.6881</td>\n",
       "      <td>14.402</td>\n",
       "      <td>3.520</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age, internal_knots=[25, 40, 60])[5]</th>\n",
       "      <td>16.6061</td>\n",
       "      <td>19.126</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.385</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            coef  std err      t  P>|t|\n",
       "intercept                                60.4937    9.460  6.394  0.000\n",
       "bs(age, internal_knots=[25, 40, 60])[0]   3.9805   12.538  0.317  0.751\n",
       "bs(age, internal_knots=[25, 40, 60])[1]  44.6310    9.626  4.636  0.000\n",
       "bs(age, internal_knots=[25, 40, 60])[2]  62.8388   10.755  5.843  0.000\n",
       "bs(age, internal_knots=[25, 40, 60])[3]  55.9908   10.706  5.230  0.000\n",
       "bs(age, internal_knots=[25, 40, 60])[4]  50.6881   14.402  3.520  0.000\n",
       "bs(age, internal_knots=[25, 40, 60])[5]  16.6061   19.126  0.868  0.385"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bs_age = MS([bs('age', internal_knots=[25,40,60])])\n",
    "Xbs = bs_age.fit_transform(Wage)\n",
    "M = sm.OLS(y, Xbs).fit()\n",
    "summarize(M)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54bacda5",
   "metadata": {},
   "source": [
    "The column names are a little cumbersome, and have caused us to truncate the printed summary. They can be set on construction using the `name` argument as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d01c2c26",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.384846Z",
     "iopub.status.busy": "2024-06-04T23:19:36.384459Z",
     "iopub.status.idle": "2024-06-04T23:19:36.418050Z",
     "shell.execute_reply": "2024-06-04T23:19:36.415940Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>60.4937</td>\n",
       "      <td>9.460</td>\n",
       "      <td>6.394</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age)[0]</th>\n",
       "      <td>3.9805</td>\n",
       "      <td>12.538</td>\n",
       "      <td>0.317</td>\n",
       "      <td>0.751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age)[1]</th>\n",
       "      <td>44.6310</td>\n",
       "      <td>9.626</td>\n",
       "      <td>4.636</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age)[2]</th>\n",
       "      <td>62.8388</td>\n",
       "      <td>10.755</td>\n",
       "      <td>5.843</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age)[3]</th>\n",
       "      <td>55.9908</td>\n",
       "      <td>10.706</td>\n",
       "      <td>5.230</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age)[4]</th>\n",
       "      <td>50.6881</td>\n",
       "      <td>14.402</td>\n",
       "      <td>3.520</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age)[5]</th>\n",
       "      <td>16.6061</td>\n",
       "      <td>19.126</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.385</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               coef  std err      t  P>|t|\n",
       "intercept   60.4937    9.460  6.394  0.000\n",
       "bs(age)[0]   3.9805   12.538  0.317  0.751\n",
       "bs(age)[1]  44.6310    9.626  4.636  0.000\n",
       "bs(age)[2]  62.8388   10.755  5.843  0.000\n",
       "bs(age)[3]  55.9908   10.706  5.230  0.000\n",
       "bs(age)[4]  50.6881   14.402  3.520  0.000\n",
       "bs(age)[5]  16.6061   19.126  0.868  0.385"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bs_age = MS([bs('age',\n",
    "                internal_knots=[25,40,60],\n",
    "                name='bs(age)')])\n",
    "Xbs = bs_age.fit_transform(Wage)\n",
    "M = sm.OLS(y, Xbs).fit()\n",
    "summarize(M)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b7fab9b",
   "metadata": {},
   "source": [
    "Notice that there are 6 spline coefficients rather than 7. This is because, by default,\n",
    "`bs()` assumes `intercept=False`, since we typically have an overall intercept in the model.\n",
    "So it generates the spline basis with the given knots,  and then discards one of the basis functions to account for the intercept. \n",
    "\n",
    "We could also use the `df` (degrees of freedom) option to\n",
    "specify the complexity of the spline.  We see above that with 3 knots,\n",
    "the spline basis has 6 columns or degrees of freedom.  When we specify\n",
    "`df=6` rather than the actual knots, `bs()` will produce a\n",
    "spline with 3 knots chosen at uniform quantiles of the training data.\n",
    "We can see these chosen knots most easily using `Bspline()` directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "672d44ae",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.421424Z",
     "iopub.status.busy": "2024-06-04T23:19:36.421207Z",
     "iopub.status.idle": "2024-06-04T23:19:36.429792Z",
     "shell.execute_reply": "2024-06-04T23:19:36.429086Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([33.75, 42.  , 51.  ])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "BSpline(df=6).fit(age).internal_knots_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57fe2029",
   "metadata": {},
   "source": [
    " When asking for six degrees of freedom,\n",
    "the transform chooses knots at ages 33.75, 42.0, and 51.0,\n",
    "which correspond to the 25th, 50th, and 75th percentiles of\n",
    "`age`.\n",
    "\n",
    "When using B-splines we need not limit ourselves to cubic polynomials\n",
    "(i.e. `degree=3`). For instance, using `degree=0` results\n",
    "in piecewise constant functions, as in our example with\n",
    "`pd.qcut()` above.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4a3f93ee",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.432272Z",
     "iopub.status.busy": "2024-06-04T23:19:36.432078Z",
     "iopub.status.idle": "2024-06-04T23:19:36.457934Z",
     "shell.execute_reply": "2024-06-04T23:19:36.457375Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>94.1584</td>\n",
       "      <td>1.478</td>\n",
       "      <td>63.687</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age, df=3, degree=0)[0]</th>\n",
       "      <td>22.3490</td>\n",
       "      <td>2.152</td>\n",
       "      <td>10.388</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age, df=3, degree=0)[1]</th>\n",
       "      <td>24.8076</td>\n",
       "      <td>2.044</td>\n",
       "      <td>12.137</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bs(age, df=3, degree=0)[2]</th>\n",
       "      <td>22.7814</td>\n",
       "      <td>2.087</td>\n",
       "      <td>10.917</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               coef  std err       t  P>|t|\n",
       "intercept                   94.1584    1.478  63.687    0.0\n",
       "bs(age, df=3, degree=0)[0]  22.3490    2.152  10.388    0.0\n",
       "bs(age, df=3, degree=0)[1]  24.8076    2.044  12.137    0.0\n",
       "bs(age, df=3, degree=0)[2]  22.7814    2.087  10.917    0.0"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bs_age0 = MS([bs('age',\n",
    "                 df=3, \n",
    "                 degree=0)]).fit(Wage)\n",
    "Xbs0 = bs_age0.transform(Wage)\n",
    "summarize(sm.OLS(y, Xbs0).fit())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad6376f5",
   "metadata": {},
   "source": [
    "This fit should be compared with cell [15] where we use `qcut()`\n",
    "to create four bins by cutting at the 25%, 50% and 75% quantiles of\n",
    "`age`.  Since we specified `df=3` for degree-zero splines here, there will also be\n",
    "knots at the same three quantiles. Although the coefficients appear different, we\n",
    "see that this is a result of the different coding. For example, the\n",
    "first coefficient is identical in both cases, and is the mean response\n",
    "in the first bin. For the second coefficient, we have\n",
    "$94.158 + 22.349 = 116.507 \\approx 116.611$, the latter being the mean\n",
    "in the second bin in cell [15]. Here the intercept is coded by a column\n",
    "of ones, so the second, third and fourth coefficients are increments\n",
    "for those bins. Why is the sum not exactly the same? It turns out that\n",
    "the `qcut()` uses $\\leq$, while `bs()` uses $<$ when\n",
    "deciding bin membership.\n",
    "\n",
    "\n",
    "    \n",
    " \n",
    "\n",
    " \n",
    "\n",
    "    \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99749b1e",
   "metadata": {},
   "source": [
    "In order to fit a natural spline, we use the `NaturalSpline()` \n",
    "transform with the corresponding helper `ns()`.  Here we fit a natural spline with five\n",
    "degrees of freedom (excluding the intercept) and plot the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "bd2b4215",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.460525Z",
     "iopub.status.busy": "2024-06-04T23:19:36.460295Z",
     "iopub.status.idle": "2024-06-04T23:19:36.489739Z",
     "shell.execute_reply": "2024-06-04T23:19:36.489248Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>60.4752</td>\n",
       "      <td>4.708</td>\n",
       "      <td>12.844</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ns(age, df=5)[0]</th>\n",
       "      <td>61.5267</td>\n",
       "      <td>4.709</td>\n",
       "      <td>13.065</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ns(age, df=5)[1]</th>\n",
       "      <td>55.6912</td>\n",
       "      <td>5.717</td>\n",
       "      <td>9.741</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ns(age, df=5)[2]</th>\n",
       "      <td>46.8184</td>\n",
       "      <td>4.948</td>\n",
       "      <td>9.463</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ns(age, df=5)[3]</th>\n",
       "      <td>83.2036</td>\n",
       "      <td>11.918</td>\n",
       "      <td>6.982</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ns(age, df=5)[4]</th>\n",
       "      <td>6.8770</td>\n",
       "      <td>9.484</td>\n",
       "      <td>0.725</td>\n",
       "      <td>0.468</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     coef  std err       t  P>|t|\n",
       "intercept         60.4752    4.708  12.844  0.000\n",
       "ns(age, df=5)[0]  61.5267    4.709  13.065  0.000\n",
       "ns(age, df=5)[1]  55.6912    5.717   9.741  0.000\n",
       "ns(age, df=5)[2]  46.8184    4.948   9.463  0.000\n",
       "ns(age, df=5)[3]  83.2036   11.918   6.982  0.000\n",
       "ns(age, df=5)[4]   6.8770    9.484   0.725  0.468"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ns_age = MS([ns('age', df=5)]).fit(Wage)\n",
    "M_ns = sm.OLS(y, ns_age.transform(Wage)).fit()\n",
    "summarize(M_ns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5bc0531",
   "metadata": {},
   "source": [
    "We now plot the natural spline using our plotting function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "9162a058",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.493122Z",
     "iopub.status.busy": "2024-06-04T23:19:36.492786Z",
     "iopub.status.idle": "2024-06-04T23:19:36.668184Z",
     "shell.execute_reply": "2024-06-04T23:19:36.667651Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 178524 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_wage_fit(age_df,\n",
    "              ns_age,\n",
    "              'Natural spline, df=5');\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb85535d",
   "metadata": {},
   "source": [
    "## Smoothing Splines and GAMs\n",
    "A smoothing spline is a special case of a GAM with squared-error loss\n",
    "and a single feature. To fit GAMs in `Python` we will use the\n",
    "`pygam` package which can be installed via `pip install pygam`. The\n",
    "estimator `LinearGAM()` uses squared-error loss.\n",
    "The GAM is specified by associating each column\n",
    "of a model matrix with a particular smoothing operation:\n",
    "`s` for smoothing spline; `l` for linear, and `f` for factor or categorical variables.\n",
    "The argument `0` passed to `s` below indicates that this smoother will\n",
    "apply to the first column of a feature matrix. Below, we pass it a\n",
    "matrix with a single column: `X_age`. The argument `lam` is the penalty parameter $\\lambda$ as discussed in Section~\\ref{Ch7:sec5.2}."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "86ff9343",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.671284Z",
     "iopub.status.busy": "2024-06-04T23:19:36.671050Z",
     "iopub.status.idle": "2024-06-04T23:19:36.705757Z",
     "shell.execute_reply": "2024-06-04T23:19:36.705247Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearGAM(callbacks=[Deviance(), Diffs()], fit_intercept=True, \n",
       "   max_iter=100, scale=None, terms=s(0) + intercept, tol=0.0001, \n",
       "   verbose=False)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_age = np.asarray(age).reshape((-1,1))\n",
    "gam = LinearGAM(s_gam(0, lam=0.6))\n",
    "gam.fit(X_age, y)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "879ee931",
   "metadata": {},
   "source": [
    "The `pygam` library generally expects a matrix of features so we reshape `age` to be a matrix (a two-dimensional array) instead\n",
    "of a vector (i.e. a one-dimensional array). The `-1` in the call to the `reshape()` method tells `numpy` to impute the\n",
    "size of that dimension based on the remaining entries of the shape tuple.\n",
    " \n",
    "Let’s investigate how the fit changes with the smoothing parameter `lam`.\n",
    "The function `np.logspace()` is similar to `np.linspace()` but spaces points\n",
    "evenly on the log-scale. Below we vary `lam` from $10^{-2}$ to $10^6$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "44812968",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.708880Z",
     "iopub.status.busy": "2024-06-04T23:19:36.708673Z",
     "iopub.status.idle": "2024-06-04T23:19:36.976882Z",
     "shell.execute_reply": "2024-06-04T23:19:36.976289Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 179744 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = subplots(figsize=(8,8))\n",
    "ax.scatter(age, y, facecolor='gray', alpha=0.5)\n",
    "for lam in np.logspace(-2, 6, 5):\n",
    "    gam = LinearGAM(s_gam(0, lam=lam)).fit(X_age, y)\n",
    "    ax.plot(age_grid,\n",
    "            gam.predict(age_grid),\n",
    "            label='{:.1e}'.format(lam),\n",
    "            linewidth=3)\n",
    "ax.set_xlabel('Age', fontsize=20)\n",
    "ax.set_ylabel('Wage', fontsize=20);\n",
    "ax.legend(title='$\\lambda$');\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f18b58db",
   "metadata": {},
   "source": [
    "The `pygam` package can perform a search for an optimal smoothing parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "968e1e42",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:36.979615Z",
     "iopub.status.busy": "2024-06-04T23:19:36.979395Z",
     "iopub.status.idle": "2024-06-04T23:19:37.195601Z",
     "shell.execute_reply": "2024-06-04T23:19:37.195005Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[38;2;255;0;0m  0%\u001b[39m \u001b[38;2;255;0;0m(0 of 11)\u001b[39m |                         | Elapsed Time: 0:00:00 ETA:  --:--:--"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[38;2;255;189;0m 45%\u001b[39m \u001b[38;2;255;189;0m(5 of 11)\u001b[39m |###########              | Elapsed Time: 0:00:00 ETA:   0:00:00"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[38;2;0;255;0m100%\u001b[39m \u001b[38;2;0;255;0m(11 of 11)\u001b[39m |########################| Elapsed Time: 0:00:00 Time:  0:00:00"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 181072 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gam_opt = gam.gridsearch(X_age, y)\n",
    "ax.plot(age_grid,\n",
    "        gam_opt.predict(age_grid),\n",
    "        label='Grid search',\n",
    "        linewidth=4)\n",
    "ax.legend()\n",
    "fig\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bba7cf02",
   "metadata": {},
   "source": [
    "Alternatively, we can fix the degrees of freedom of the smoothing\n",
    "spline using a function included in the `ISLP.pygam` package. Below we\n",
    "find a value of $\\lambda$ that gives us roughly four degrees of\n",
    "freedom. We note here that these degrees of freedom include the\n",
    "unpenalized intercept and linear term of the smoothing spline, hence there are at least two\n",
    "degrees of freedom."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "891e3b75",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:37.198482Z",
     "iopub.status.busy": "2024-06-04T23:19:37.198260Z",
     "iopub.status.idle": "2024-06-04T23:19:37.214885Z",
     "shell.execute_reply": "2024-06-04T23:19:37.214249Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.000000100001664"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "age_term = gam.terms[0]\n",
    "lam_4 = approx_lam(X_age, age_term, 4)\n",
    "age_term.lam = lam_4\n",
    "degrees_of_freedom(X_age, age_term)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7688d014",
   "metadata": {},
   "source": [
    "Let’s vary the degrees of freedom in a similar plot to above. We choose the degrees of freedom\n",
    "as the desired degrees of freedom plus one to account for the fact that these smoothing\n",
    "splines always have an intercept term. Hence, a value of one for `df` is just a linear fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "54715029",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:37.218081Z",
     "iopub.status.busy": "2024-06-04T23:19:37.217683Z",
     "iopub.status.idle": "2024-06-04T23:19:37.501731Z",
     "shell.execute_reply": "2024-06-04T23:19:37.500504Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 185816 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = subplots(figsize=(8,8))\n",
    "ax.scatter(X_age,\n",
    "           y,\n",
    "           facecolor='gray',\n",
    "           alpha=0.3)\n",
    "for df in [1,3,4,8,15]:\n",
    "    lam = approx_lam(X_age, age_term, df+1)\n",
    "    age_term.lam = lam\n",
    "    gam.fit(X_age, y)\n",
    "    ax.plot(age_grid,\n",
    "            gam.predict(age_grid),\n",
    "            label='{:d}'.format(df),\n",
    "            linewidth=4)\n",
    "ax.set_xlabel('Age', fontsize=20)\n",
    "ax.set_ylabel('Wage', fontsize=20);\n",
    "ax.legend(title='Degrees of freedom');\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cec1eac",
   "metadata": {},
   "source": [
    "### Additive Models with Several Terms\n",
    "The strength of generalized additive models lies in their ability to fit multivariate regression models with more flexibility than linear models. We demonstrate two approaches: the first in a more manual fashion using natural splines and piecewise constant functions, and the second  using the `pygam` package and smoothing splines.\n",
    "\n",
    "We now fit a GAM by hand to predict\n",
    "`wage` using natural spline functions of `year` and `age`,\n",
    "treating `education` as a qualitative predictor, as in (\\ref{Ch7:nsmod}).\n",
    "Since this is just a big linear regression model\n",
    "using an appropriate choice of basis functions, we can simply do this\n",
    "using the `sm.OLS()`  function.\n",
    "\n",
    "We will build the model matrix in a more manual fashion here, since we wish\n",
    "to access the pieces separately when constructing partial dependence plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "b3cd8b66",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:37.505073Z",
     "iopub.status.busy": "2024-06-04T23:19:37.504839Z",
     "iopub.status.idle": "2024-06-04T23:19:37.519804Z",
     "shell.execute_reply": "2024-06-04T23:19:37.519297Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "ns_age = NaturalSpline(df=4).fit(age)\n",
    "ns_year = NaturalSpline(df=5).fit(Wage['year'])\n",
    "Xs = [ns_age.transform(age),\n",
    "      ns_year.transform(Wage['year']),\n",
    "      pd.get_dummies(Wage['education']).values]\n",
    "X_bh = np.hstack(Xs)\n",
    "gam_bh = sm.OLS(y, X_bh).fit()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a1668c8",
   "metadata": {},
   "source": [
    "Here the function `NaturalSpline()` is the workhorse supporting\n",
    "the `ns()` helper function.  We chose to use all columns of the\n",
    "indicator matrix for the categorical variable `education`, making an intercept redundant.\n",
    "Finally, we stacked the three component matrices horizontally to form the model matrix `X_bh`. \n",
    "\n",
    "We now show how to construct partial dependence plots for each of the terms in our rudimentary GAM. We can do this by hand,\n",
    "given grids for `age` and `year`.\n",
    " We simply predict with new $X$ matrices, fixing all but one of the features at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "ea8d6bc5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:37.523112Z",
     "iopub.status.busy": "2024-06-04T23:19:37.522834Z",
     "iopub.status.idle": "2024-06-04T23:19:37.644337Z",
     "shell.execute_reply": "2024-06-04T23:19:37.643251Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 66320 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "age_grid = np.linspace(age.min(),\n",
    "                       age.max(),\n",
    "                       100)\n",
    "X_age_bh = X_bh.copy()[:100]\n",
    "X_age_bh[:] = X_bh[:].mean(0)[None,:]\n",
    "X_age_bh[:,:4] = ns_age.transform(age_grid)\n",
    "preds = gam_bh.get_prediction(X_age_bh)\n",
    "bounds_age = preds.conf_int(alpha=0.05)\n",
    "partial_age = preds.predicted_mean\n",
    "center = partial_age.mean()\n",
    "partial_age -= center\n",
    "bounds_age -= center\n",
    "fig, ax = subplots(figsize=(8,8))\n",
    "ax.plot(age_grid, partial_age, 'b', linewidth=3)\n",
    "ax.plot(age_grid, bounds_age[:,0], 'r--', linewidth=3)\n",
    "ax.plot(age_grid, bounds_age[:,1], 'r--', linewidth=3)\n",
    "ax.set_xlabel('Age')\n",
    "ax.set_ylabel('Effect on wage')\n",
    "ax.set_title('Partial dependence of age on wage', fontsize=20);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ae3a3bb",
   "metadata": {},
   "source": [
    "Let's explain in some detail what we did above. The idea is to create a new prediction matrix, where all but the columns belonging to `age` are constant (and set to  their training-data means). The four columns for `age` are filled in with the natural spline basis evaluated at the 100 values in `age_grid`.\n",
    "\n",
    "* We made a grid of length 100 in `age`, and created a matrix `X_age_bh` with 100 rows and the same number of columns as `X_bh`.\n",
    "* We replaced every row of this matrix with the column means of the original.\n",
    "* We then replace just the first four columns representing `age` with the natural spline basis computed at the values in `age_grid`. \n",
    "\n",
    "The remaining steps should by now be familiar.\n",
    "\n",
    "We also look at the effect of `year` on `wage`; the process is the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "812a47a1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:37.648530Z",
     "iopub.status.busy": "2024-06-04T23:19:37.647173Z",
     "iopub.status.idle": "2024-06-04T23:19:37.756140Z",
     "shell.execute_reply": "2024-06-04T23:19:37.755581Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 63176 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "year_grid = np.linspace(2003, 2009, 100)\n",
    "year_grid = np.linspace(Wage['year'].min(),\n",
    "                        Wage['year'].max(),\n",
    "                        100)\n",
    "X_year_bh = X_bh.copy()[:100]\n",
    "X_year_bh[:] = X_bh[:].mean(0)[None,:]\n",
    "X_year_bh[:,4:9] = ns_year.transform(year_grid)\n",
    "preds = gam_bh.get_prediction(X_year_bh)\n",
    "bounds_year = preds.conf_int(alpha=0.05)\n",
    "partial_year = preds.predicted_mean\n",
    "center = partial_year.mean()\n",
    "partial_year -= center\n",
    "bounds_year -= center\n",
    "fig, ax = subplots(figsize=(8,8))\n",
    "ax.plot(year_grid, partial_year, 'b', linewidth=3)\n",
    "ax.plot(year_grid, bounds_year[:,0], 'r--', linewidth=3)\n",
    "ax.plot(year_grid, bounds_year[:,1], 'r--', linewidth=3)\n",
    "ax.set_xlabel('Year')\n",
    "ax.set_ylabel('Effect on wage')\n",
    "ax.set_title('Partial dependence of year on wage', fontsize=20);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cd84b43",
   "metadata": {},
   "source": [
    "We now fit the model (\\ref{Ch7:nsmod})  using smoothing splines rather\n",
    "than natural splines.  All of the\n",
    "terms in  (\\ref{Ch7:nsmod})  are fit simultaneously, taking each other\n",
    "into account to explain the response. The `pygam` package only works with matrices, so we must convert\n",
    "the categorical series `education` to its array representation, which can be found\n",
    "with the `cat.codes` attribute of `education`. As `year` only has 7 unique values, we\n",
    "use only seven basis functions for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "88ff3def",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:37.759390Z",
     "iopub.status.busy": "2024-06-04T23:19:37.759130Z",
     "iopub.status.idle": "2024-06-04T23:19:37.788899Z",
     "shell.execute_reply": "2024-06-04T23:19:37.788370Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "gam_full = LinearGAM(s_gam(0) +\n",
    "                     s_gam(1, n_splines=7) +\n",
    "                     f_gam(2, lam=0))\n",
    "Xgam = np.column_stack([age,\n",
    "                        Wage['year'],\n",
    "                        Wage['education'].cat.codes])\n",
    "gam_full = gam_full.fit(Xgam, y)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eedda235",
   "metadata": {},
   "source": [
    "The two `s_gam()` terms result in smoothing spline fits, and use a default value for $\\lambda$  (`lam=0.6`), which is somewhat arbitrary. For the categorical term `education`, specified using a `f_gam()` term,  we specify `lam=0` to avoid any shrinkage.\n",
    "We produce the partial dependence plot in `age` to see the effect of these choices.\n",
    "\n",
    "The values for the plot\n",
    "are generated by the `pygam` package. We provide a `plot_gam()`\n",
    "function for partial-dependence plots in `ISLP.pygam`, which makes this job easier than in our last example with natural splines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "29dbc933",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:37.792129Z",
     "iopub.status.busy": "2024-06-04T23:19:37.791875Z",
     "iopub.status.idle": "2024-06-04T23:19:37.909465Z",
     "shell.execute_reply": "2024-06-04T23:19:37.908764Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 74424 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = subplots(figsize=(8,8))\n",
    "plot_gam(gam_full, 0, ax=ax)\n",
    "ax.set_xlabel('Age')\n",
    "ax.set_ylabel('Effect on wage')\n",
    "ax.set_title('Partial dependence of age on wage - default lam=0.6', fontsize=20);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcaa9a68",
   "metadata": {},
   "source": [
    "We see that the function is somewhat wiggly. It is more natural to specify the `df` than a value for `lam`. \n",
    "We refit a GAM using four degrees of freedom each for\n",
    "`age` and  `year`. Recall that the addition of one below takes into account the intercept\n",
    "of the smoothing spline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "cdb535b5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:37.912957Z",
     "iopub.status.busy": "2024-06-04T23:19:37.912683Z",
     "iopub.status.idle": "2024-06-04T23:19:37.946458Z",
     "shell.execute_reply": "2024-06-04T23:19:37.945953Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "age_term = gam_full.terms[0]\n",
    "age_term.lam = approx_lam(Xgam, age_term, df=4+1)\n",
    "year_term = gam_full.terms[1]\n",
    "year_term.lam = approx_lam(Xgam, year_term, df=4+1)\n",
    "gam_full = gam_full.fit(Xgam, y)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6a22de8",
   "metadata": {},
   "source": [
    "Note that updating `age_term.lam` above updates it in `gam_full.terms[0]` as well! Likewise for `year_term.lam`.\n",
    "\n",
    "Repeating the plot for `age`, we see that it is much smoother.\n",
    "We also produce the plot for `year`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "2acadc27",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:37.950913Z",
     "iopub.status.busy": "2024-06-04T23:19:37.950529Z",
     "iopub.status.idle": "2024-06-04T23:19:38.082439Z",
     "shell.execute_reply": "2024-06-04T23:19:38.081725Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Partial dependence of year on wage')"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 60368 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = subplots(figsize=(8,8))\n",
    "plot_gam(gam_full,\n",
    "         1,\n",
    "         ax=ax)\n",
    "ax.set_xlabel('Year')\n",
    "ax.set_ylabel('Effect on wage')\n",
    "ax.set_title('Partial dependence of year on wage', fontsize=20)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16258b84",
   "metadata": {},
   "source": [
    "Finally we plot `education`, which is categorical. The partial dependence plot is different, and more suitable for the set of fitted constants for each level of this variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "a2b8e481",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.085348Z",
     "iopub.status.busy": "2024-06-04T23:19:38.085138Z",
     "iopub.status.idle": "2024-06-04T23:19:38.186025Z",
     "shell.execute_reply": "2024-06-04T23:19:38.185091Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 36540 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = subplots(figsize=(8, 8))\n",
    "ax = plot_gam(gam_full, 2)\n",
    "ax.set_xlabel('Education')\n",
    "ax.set_ylabel('Effect on wage')\n",
    "ax.set_title('Partial dependence of wage on education',\n",
    "             fontsize=20);\n",
    "ax.set_xticklabels(Wage['education'].cat.categories, fontsize=8);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a21bb76",
   "metadata": {},
   "source": [
    "### ANOVA Tests for Additive Models\n",
    "In all of our models, the function of `year` looks rather linear. We can\n",
    "perform a series of ANOVA tests in order to determine which of these\n",
    "three models is best: a GAM that excludes `year` ($\\mathcal{M}_1$),\n",
    "a GAM that uses a linear function of `year` ($\\mathcal{M}_2$), or a\n",
    "GAM that uses a spline function of `year` ($\\mathcal{M}_3$)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "1a76b516",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.189290Z",
     "iopub.status.busy": "2024-06-04T23:19:38.188978Z",
     "iopub.status.idle": "2024-06-04T23:19:38.226707Z",
     "shell.execute_reply": "2024-06-04T23:19:38.226124Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearGAM(callbacks=[Deviance(), Diffs()], fit_intercept=True, \n",
       "   max_iter=100, scale=None, terms=s(0) + l(1) + f(2) + intercept, \n",
       "   tol=0.0001, verbose=False)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gam_0 = LinearGAM(age_term + f_gam(2, lam=0))\n",
    "gam_0.fit(Xgam, y)\n",
    "gam_linear = LinearGAM(age_term +\n",
    "                       l_gam(1, lam=0) +\n",
    "                       f_gam(2, lam=0))\n",
    "gam_linear.fit(Xgam, y)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0de32e6",
   "metadata": {},
   "source": [
    "Notice our use of `age_term` in the expressions above. We do this because\n",
    "earlier we set the value for `lam` in this term to achieve four degrees of freedom.\n",
    "\n",
    "To directly assess the effect of `year` we run an ANOVA on the\n",
    "three models fit above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "7a68bf04",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.229152Z",
     "iopub.status.busy": "2024-06-04T23:19:38.228941Z",
     "iopub.status.idle": "2024-06-04T23:19:38.236919Z",
     "shell.execute_reply": "2024-06-04T23:19:38.236171Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>deviance</th>\n",
       "      <th>df</th>\n",
       "      <th>deviance_diff</th>\n",
       "      <th>df_diff</th>\n",
       "      <th>F</th>\n",
       "      <th>pvalue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.714362e+06</td>\n",
       "      <td>2991.004005</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.696746e+06</td>\n",
       "      <td>2990.005190</td>\n",
       "      <td>17616.542840</td>\n",
       "      <td>0.998815</td>\n",
       "      <td>14.265131</td>\n",
       "      <td>0.002314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.693143e+06</td>\n",
       "      <td>2987.007254</td>\n",
       "      <td>3602.893655</td>\n",
       "      <td>2.997936</td>\n",
       "      <td>0.972007</td>\n",
       "      <td>0.435579</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       deviance           df  deviance_diff   df_diff          F    pvalue\n",
       "0  3.714362e+06  2991.004005            NaN       NaN        NaN       NaN\n",
       "1  3.696746e+06  2990.005190   17616.542840  0.998815  14.265131  0.002314\n",
       "2  3.693143e+06  2987.007254    3602.893655  2.997936   0.972007  0.435579"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anova_gam(gam_0, gam_linear, gam_full)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed03248f",
   "metadata": {},
   "source": [
    " We find that there is compelling evidence that a GAM with a linear\n",
    "function in `year` is better than a GAM that does not include\n",
    "`year` at all ($p$-value= 0.002). However, there is no\n",
    "evidence that a non-linear function of `year` is needed\n",
    "($p$-value=0.435).  In other words, based on the results of this\n",
    "ANOVA, $\\mathcal{M}_2$ is preferred.\n",
    "\n",
    "We can repeat the same process for `age` as well. We see there is very clear evidence that\n",
    "a non-linear term is required for `age`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "a05df53b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.239488Z",
     "iopub.status.busy": "2024-06-04T23:19:38.239277Z",
     "iopub.status.idle": "2024-06-04T23:19:38.264822Z",
     "shell.execute_reply": "2024-06-04T23:19:38.264344Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>deviance</th>\n",
       "      <th>df</th>\n",
       "      <th>deviance_diff</th>\n",
       "      <th>df_diff</th>\n",
       "      <th>F</th>\n",
       "      <th>pvalue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.975443e+06</td>\n",
       "      <td>2991.000589</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.850247e+06</td>\n",
       "      <td>2990.000704</td>\n",
       "      <td>125196.137317</td>\n",
       "      <td>0.999884</td>\n",
       "      <td>101.270106</td>\n",
       "      <td>1.681120e-07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.693143e+06</td>\n",
       "      <td>2987.007254</td>\n",
       "      <td>157103.978302</td>\n",
       "      <td>2.993450</td>\n",
       "      <td>42.447812</td>\n",
       "      <td>5.669414e-07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       deviance           df  deviance_diff   df_diff           F  \\\n",
       "0  3.975443e+06  2991.000589            NaN       NaN         NaN   \n",
       "1  3.850247e+06  2990.000704  125196.137317  0.999884  101.270106   \n",
       "2  3.693143e+06  2987.007254  157103.978302  2.993450   42.447812   \n",
       "\n",
       "         pvalue  \n",
       "0           NaN  \n",
       "1  1.681120e-07  \n",
       "2  5.669414e-07  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gam_0 = LinearGAM(year_term +\n",
    "                  f_gam(2, lam=0))\n",
    "gam_linear = LinearGAM(l_gam(0, lam=0) +\n",
    "                       year_term +\n",
    "                       f_gam(2, lam=0))\n",
    "gam_0.fit(Xgam, y)\n",
    "gam_linear.fit(Xgam, y)\n",
    "anova_gam(gam_0, gam_linear, gam_full)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bf66bc5",
   "metadata": {},
   "source": [
    "There is a (verbose) `summary()` method for the GAM fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "ee9db82c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.267342Z",
     "iopub.status.busy": "2024-06-04T23:19:38.267144Z",
     "iopub.status.idle": "2024-06-04T23:19:38.271357Z",
     "shell.execute_reply": "2024-06-04T23:19:38.270406Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LinearGAM                                                                                                 \n",
      "=============================================== ==========================================================\n",
      "Distribution:                        NormalDist Effective DoF:                                     12.9927\n",
      "Link Function:                     IdentityLink Log Likelihood:                                 -24117.907\n",
      "Number of Samples:                         3000 AIC:                                            48263.7995\n",
      "                                                AICc:                                             48263.94\n",
      "                                                GCV:                                             1246.1129\n",
      "                                                Scale:                                           1236.4024\n",
      "                                                Pseudo R-Squared:                                   0.2928\n",
      "==========================================================================================================\n",
      "Feature Function                  Lambda               Rank         EDoF         P > x        Sig. Code   \n",
      "================================= ==================== ============ ============ ============ ============\n",
      "s(0)                              [465.0491]           20           5.1          1.11e-16     ***         \n",
      "s(1)                              [2.1564]             7            4.0          8.10e-03     **          \n",
      "f(2)                              [0]                  5            4.0          1.11e-16     ***         \n",
      "intercept                                              1            0.0          1.11e-16     ***         \n",
      "==========================================================================================================\n",
      "Significance codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n",
      "\n",
      "WARNING: Fitting splines and a linear function to a feature introduces a model identifiability problem\n",
      "         which can cause p-values to appear significant when they are not.\n",
      "\n",
      "WARNING: p-values calculated in this manner behave correctly for un-penalized models or models with\n",
      "         known smoothing parameters, but when smoothing parameters have been estimated, the p-values\n",
      "         are typically lower than they should be, meaning that the tests reject the null too readily.\n"
     ]
    }
   ],
   "source": [
    "gam_full.summary()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fbf126f",
   "metadata": {},
   "source": [
    "We can make predictions from `gam` objects, just like from\n",
    "`lm` objects, using the `predict()`  method for the class\n",
    "`gam`.  Here we make predictions on the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "cf0a024d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.274135Z",
     "iopub.status.busy": "2024-06-04T23:19:38.273728Z",
     "iopub.status.idle": "2024-06-04T23:19:38.283548Z",
     "shell.execute_reply": "2024-06-04T23:19:38.283095Z"
    }
   },
   "outputs": [],
   "source": [
    "Yhat = gam_full.predict(Xgam)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70afdc16",
   "metadata": {},
   "source": [
    "In order to fit a logistic regression GAM, we use `LogisticGAM()` \n",
    "from `pygam`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "0fe9f031",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.286193Z",
     "iopub.status.busy": "2024-06-04T23:19:38.285994Z",
     "iopub.status.idle": "2024-06-04T23:19:38.372533Z",
     "shell.execute_reply": "2024-06-04T23:19:38.372014Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticGAM(callbacks=[Deviance(), Diffs(), Accuracy()], \n",
       "   fit_intercept=True, max_iter=100, \n",
       "   terms=s(0) + l(1) + f(2) + intercept, tol=0.0001, verbose=False)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gam_logit = LogisticGAM(age_term + \n",
    "                        l_gam(1, lam=0) +\n",
    "                        f_gam(2, lam=0))\n",
    "gam_logit.fit(Xgam, high_earn)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "bdc73505",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.375490Z",
     "iopub.status.busy": "2024-06-04T23:19:38.375048Z",
     "iopub.status.idle": "2024-06-04T23:19:38.476110Z",
     "shell.execute_reply": "2024-06-04T23:19:38.475521Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 39352 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = subplots(figsize=(8, 8))\n",
    "ax = plot_gam(gam_logit, 2)\n",
    "ax.set_xlabel('Education')\n",
    "ax.set_ylabel('Effect on wage')\n",
    "ax.set_title('Partial dependence of wage on education',\n",
    "             fontsize=20);\n",
    "ax.set_xticklabels(Wage['education'].cat.categories, fontsize=8);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aed5a52",
   "metadata": {},
   "source": [
    "The model seems to be very flat, with especially high error bars for the first category.\n",
    "Let's look at the data a bit more closely."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "44f57a30",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.478694Z",
     "iopub.status.busy": "2024-06-04T23:19:38.478494Z",
     "iopub.status.idle": "2024-06-04T23:19:38.491764Z",
     "shell.execute_reply": "2024-06-04T23:19:38.491357Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>education</th>\n",
       "      <th>1. &lt; HS Grad</th>\n",
       "      <th>2. HS Grad</th>\n",
       "      <th>3. Some College</th>\n",
       "      <th>4. College Grad</th>\n",
       "      <th>5. Advanced Degree</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>high_earn</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>False</th>\n",
       "      <td>268</td>\n",
       "      <td>966</td>\n",
       "      <td>643</td>\n",
       "      <td>663</td>\n",
       "      <td>381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>True</th>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>22</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "education  1. < HS Grad  2. HS Grad  3. Some College  4. College Grad  \\\n",
       "high_earn                                                               \n",
       "False               268         966              643              663   \n",
       "True                  0           5                7               22   \n",
       "\n",
       "education  5. Advanced Degree  \n",
       "high_earn                      \n",
       "False                     381  \n",
       "True                       45  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab(Wage['high_earn'], Wage['education'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1828779e",
   "metadata": {},
   "source": [
    "We see that there are no high earners in the first category of\n",
    "education, meaning that the model will have a hard time fitting.  We\n",
    "will fit a logistic regression GAM excluding all observations falling into this\n",
    "category. This provides more sensible results.\n",
    "\n",
    "To do so,\n",
    "we could subset the model matrix, though this will not remove the\n",
    "column from `Xgam`. While we can deduce which column corresponds\n",
    "to this feature, for reproducibility’s sake we reform the model matrix\n",
    "on this smaller subset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "3b061ffd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.494311Z",
     "iopub.status.busy": "2024-06-04T23:19:38.494116Z",
     "iopub.status.idle": "2024-06-04T23:19:38.498822Z",
     "shell.execute_reply": "2024-06-04T23:19:38.498423Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "only_hs = Wage['education'] == '1. < HS Grad'\n",
    "Wage_ = Wage.loc[~only_hs]\n",
    "Xgam_ = np.column_stack([Wage_['age'],\n",
    "                         Wage_['year'],\n",
    "                         Wage_['education'].cat.codes-1])\n",
    "high_earn_ = Wage_['high_earn']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e3b8e93",
   "metadata": {},
   "source": [
    "In the second-to-last line above, we subtract one  from the codes of the category, due to a bug in `pygam`. It just relabels\n",
    "the education values and hence has no effect on the fit.\n",
    " \n",
    "We now fit the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "220e26ac",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.501236Z",
     "iopub.status.busy": "2024-06-04T23:19:38.501049Z",
     "iopub.status.idle": "2024-06-04T23:19:38.540965Z",
     "shell.execute_reply": "2024-06-04T23:19:38.540424Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticGAM(callbacks=[Deviance(), Diffs(), Accuracy()], \n",
       "   fit_intercept=True, max_iter=100, \n",
       "   terms=s(0) + s(1) + f(2) + intercept, tol=0.0001, verbose=False)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gam_logit_ = LogisticGAM(age_term +\n",
    "                         year_term +\n",
    "                         f_gam(2, lam=0))\n",
    "gam_logit_.fit(Xgam_, high_earn_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9838bbdc",
   "metadata": {},
   "source": [
    "Let’s look at the effect of `education`, `year` and `age` on high earner status now that we’ve\n",
    "removed those observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "94209d95",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.543510Z",
     "iopub.status.busy": "2024-06-04T23:19:38.543313Z",
     "iopub.status.idle": "2024-06-04T23:19:38.635421Z",
     "shell.execute_reply": "2024-06-04T23:19:38.634852Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 33092 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = subplots(figsize=(8, 8))\n",
    "ax = plot_gam(gam_logit_, 2)\n",
    "ax.set_xlabel('Education')\n",
    "ax.set_ylabel('Effect on wage')\n",
    "ax.set_title('Partial dependence of high earner status on education', fontsize=20);\n",
    "ax.set_xticklabels(Wage['education'].cat.categories[1:],\n",
    "                   fontsize=8);\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "b86a257f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.637945Z",
     "iopub.status.busy": "2024-06-04T23:19:38.637732Z",
     "iopub.status.idle": "2024-06-04T23:19:38.738536Z",
     "shell.execute_reply": "2024-06-04T23:19:38.737576Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 53180 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = subplots(figsize=(8, 8))\n",
    "ax = plot_gam(gam_logit_, 1)\n",
    "ax.set_xlabel('Year')\n",
    "ax.set_ylabel('Effect on wage')\n",
    "ax.set_title('Partial dependence of high earner status on year',\n",
    "             fontsize=20);\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "075082df",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.741522Z",
     "iopub.status.busy": "2024-06-04T23:19:38.741306Z",
     "iopub.status.idle": "2024-06-04T23:19:38.853209Z",
     "shell.execute_reply": "2024-06-04T23:19:38.852962Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 59880 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = subplots(figsize=(8, 8))\n",
    "ax = plot_gam(gam_logit_, 0)\n",
    "ax.set_xlabel('Age')\n",
    "ax.set_ylabel('Effect on wage')\n",
    "ax.set_title('Partial dependence of high earner status on age', fontsize=20);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94f2d878",
   "metadata": {},
   "source": [
    "## Local Regression\n",
    "We illustrate the use of local regression using  the `lowess()` \n",
    "function from `sm.nonparametric`. Some implementations of\n",
    "GAMs allow terms to be local regression operators; this is not the case in `pygam`.\n",
    "\n",
    "Here we fit local linear regression models using spans of 0.2\n",
    "and 0.5; that is, each neighborhood consists of 20% or 50% of\n",
    "the observations. As expected, using a span of 0.5 is smoother than 0.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "ecef97c8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:38.854567Z",
     "iopub.status.busy": "2024-06-04T23:19:38.854494Z",
     "iopub.status.idle": "2024-06-04T23:19:38.953710Z",
     "shell.execute_reply": "2024-06-04T23:19:38.953441Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 167208 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lowess = sm.nonparametric.lowess\n",
    "fig, ax = subplots(figsize=(8,8))\n",
    "ax.scatter(age, y, facecolor='gray', alpha=0.5)\n",
    "for span in [0.2, 0.5]:\n",
    "    fitted = lowess(y,\n",
    "                    age,\n",
    "                    frac=span,\n",
    "                    xvals=age_grid)\n",
    "    ax.plot(age_grid,\n",
    "            fitted,\n",
    "            label='{:.1f}'.format(span),\n",
    "            linewidth=4)\n",
    "ax.set_xlabel('Age', fontsize=20)\n",
    "ax.set_ylabel('Wage', fontsize=20);\n",
    "ax.legend(title='span', fontsize=15);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "163ef0c0",
   "metadata": {},
   "source": [
    "  \n",
    " \n"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
