{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "31b77867",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "559b7845",
   "metadata": {},
   "source": [
    "# Logistic Regression, LDA, QDA, and KNN\n",
    "\n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/intro-stat-learning/ISLP_labs/blob/v2.2/Ch04-classification-lab.ipynb\">\n",
    "<img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "\n",
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/intro-stat-learning/ISLP_labs/v2.2?labpath=Ch04-classification-lab.ipynb)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73b275ae",
   "metadata": {},
   "source": [
    "## The Stock Market Data\n",
    "\n",
    "In this lab we will examine the `Smarket` \n",
    "data, which is part of the `ISLP`\n",
    "library. This data set consists of percentage returns for the S&P 500\n",
    "stock index over 1,250 days, from the beginning of 2001 until the end\n",
    "of 2005. For each date, we have recorded the percentage returns for\n",
    "each of the five previous trading days,  `Lag1`  through\n",
    " `Lag5`. We have also recorded  `Volume`  (the number of\n",
    "shares traded on the previous day, in billions),  `Today`  (the\n",
    "percentage return on the date in question) and  `Direction`\n",
    "(whether the market was  `Up`  or  `Down`  on this date).\n",
    "\n",
    "We start by importing  our libraries at this top level; these are all imports we have seen in previous labs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "28bd64cb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:09.508409Z",
     "iopub.status.busy": "2024-06-04T23:19:09.508152Z",
     "iopub.status.idle": "2024-06-04T23:19:10.300967Z",
     "shell.execute_reply": "2024-06-04T23:19:10.300663Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from matplotlib.pyplot import subplots\n",
    "import statsmodels.api as sm\n",
    "from ISLP import load_data\n",
    "from ISLP.models import (ModelSpec as MS,\n",
    "                         summarize)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63093cb3",
   "metadata": {},
   "source": [
    "We also collect together the new imports needed for this lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f0a0173d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.302676Z",
     "iopub.status.busy": "2024-06-04T23:19:10.302557Z",
     "iopub.status.idle": "2024-06-04T23:19:10.318173Z",
     "shell.execute_reply": "2024-06-04T23:19:10.317991Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": [
    "from ISLP import confusion_table\n",
    "from ISLP.models import contrast\n",
    "from sklearn.discriminant_analysis import \\\n",
    "     (LinearDiscriminantAnalysis as LDA,\n",
    "      QuadraticDiscriminantAnalysis as QDA)\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1398b276",
   "metadata": {},
   "source": [
    "Now we are ready to load the `Smarket` data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3640fda5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.319434Z",
     "iopub.status.busy": "2024-06-04T23:19:10.319329Z",
     "iopub.status.idle": "2024-06-04T23:19:10.328090Z",
     "shell.execute_reply": "2024-06-04T23:19:10.327895Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year</th>\n",
       "      <th>Lag1</th>\n",
       "      <th>Lag2</th>\n",
       "      <th>Lag3</th>\n",
       "      <th>Lag4</th>\n",
       "      <th>Lag5</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Today</th>\n",
       "      <th>Direction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2001</td>\n",
       "      <td>0.381</td>\n",
       "      <td>-0.192</td>\n",
       "      <td>-2.624</td>\n",
       "      <td>-1.055</td>\n",
       "      <td>5.010</td>\n",
       "      <td>1.19130</td>\n",
       "      <td>0.959</td>\n",
       "      <td>Up</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2001</td>\n",
       "      <td>0.959</td>\n",
       "      <td>0.381</td>\n",
       "      <td>-0.192</td>\n",
       "      <td>-2.624</td>\n",
       "      <td>-1.055</td>\n",
       "      <td>1.29650</td>\n",
       "      <td>1.032</td>\n",
       "      <td>Up</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2001</td>\n",
       "      <td>1.032</td>\n",
       "      <td>0.959</td>\n",
       "      <td>0.381</td>\n",
       "      <td>-0.192</td>\n",
       "      <td>-2.624</td>\n",
       "      <td>1.41120</td>\n",
       "      <td>-0.623</td>\n",
       "      <td>Down</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2001</td>\n",
       "      <td>-0.623</td>\n",
       "      <td>1.032</td>\n",
       "      <td>0.959</td>\n",
       "      <td>0.381</td>\n",
       "      <td>-0.192</td>\n",
       "      <td>1.27600</td>\n",
       "      <td>0.614</td>\n",
       "      <td>Up</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2001</td>\n",
       "      <td>0.614</td>\n",
       "      <td>-0.623</td>\n",
       "      <td>1.032</td>\n",
       "      <td>0.959</td>\n",
       "      <td>0.381</td>\n",
       "      <td>1.20570</td>\n",
       "      <td>0.213</td>\n",
       "      <td>Up</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1245</th>\n",
       "      <td>2005</td>\n",
       "      <td>0.422</td>\n",
       "      <td>0.252</td>\n",
       "      <td>-0.024</td>\n",
       "      <td>-0.584</td>\n",
       "      <td>-0.285</td>\n",
       "      <td>1.88850</td>\n",
       "      <td>0.043</td>\n",
       "      <td>Up</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1246</th>\n",
       "      <td>2005</td>\n",
       "      <td>0.043</td>\n",
       "      <td>0.422</td>\n",
       "      <td>0.252</td>\n",
       "      <td>-0.024</td>\n",
       "      <td>-0.584</td>\n",
       "      <td>1.28581</td>\n",
       "      <td>-0.955</td>\n",
       "      <td>Down</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1247</th>\n",
       "      <td>2005</td>\n",
       "      <td>-0.955</td>\n",
       "      <td>0.043</td>\n",
       "      <td>0.422</td>\n",
       "      <td>0.252</td>\n",
       "      <td>-0.024</td>\n",
       "      <td>1.54047</td>\n",
       "      <td>0.130</td>\n",
       "      <td>Up</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1248</th>\n",
       "      <td>2005</td>\n",
       "      <td>0.130</td>\n",
       "      <td>-0.955</td>\n",
       "      <td>0.043</td>\n",
       "      <td>0.422</td>\n",
       "      <td>0.252</td>\n",
       "      <td>1.42236</td>\n",
       "      <td>-0.298</td>\n",
       "      <td>Down</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1249</th>\n",
       "      <td>2005</td>\n",
       "      <td>-0.298</td>\n",
       "      <td>0.130</td>\n",
       "      <td>-0.955</td>\n",
       "      <td>0.043</td>\n",
       "      <td>0.422</td>\n",
       "      <td>1.38254</td>\n",
       "      <td>-0.489</td>\n",
       "      <td>Down</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1250 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Year   Lag1   Lag2   Lag3   Lag4   Lag5   Volume  Today Direction\n",
       "0     2001  0.381 -0.192 -2.624 -1.055  5.010  1.19130  0.959        Up\n",
       "1     2001  0.959  0.381 -0.192 -2.624 -1.055  1.29650  1.032        Up\n",
       "2     2001  1.032  0.959  0.381 -0.192 -2.624  1.41120 -0.623      Down\n",
       "3     2001 -0.623  1.032  0.959  0.381 -0.192  1.27600  0.614        Up\n",
       "4     2001  0.614 -0.623  1.032  0.959  0.381  1.20570  0.213        Up\n",
       "...    ...    ...    ...    ...    ...    ...      ...    ...       ...\n",
       "1245  2005  0.422  0.252 -0.024 -0.584 -0.285  1.88850  0.043        Up\n",
       "1246  2005  0.043  0.422  0.252 -0.024 -0.584  1.28581 -0.955      Down\n",
       "1247  2005 -0.955  0.043  0.422  0.252 -0.024  1.54047  0.130        Up\n",
       "1248  2005  0.130 -0.955  0.043  0.422  0.252  1.42236 -0.298      Down\n",
       "1249  2005 -0.298  0.130 -0.955  0.043  0.422  1.38254 -0.489      Down\n",
       "\n",
       "[1250 rows x 9 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Smarket = load_data('Smarket')\n",
    "Smarket"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f40d5bbb",
   "metadata": {},
   "source": [
    "This gives a truncated listing of the data.\n",
    "We can see what the variable names are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ff825fa5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.329300Z",
     "iopub.status.busy": "2024-06-04T23:19:10.329214Z",
     "iopub.status.idle": "2024-06-04T23:19:10.331118Z",
     "shell.execute_reply": "2024-06-04T23:19:10.330931Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Year', 'Lag1', 'Lag2', 'Lag3', 'Lag4', 'Lag5', 'Volume', 'Today',\n",
       "       'Direction'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Smarket.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c87b92f",
   "metadata": {},
   "source": [
    "We compute the correlation matrix using the `corr()` method\n",
    "for data frames, which produces a matrix that contains all of\n",
    "the pairwise correlations among the variables.\n",
    " \n",
    "By instructing `pandas` to use only numeric variables, the `corr()` method does not report a correlation for the `Direction`  variable because it is\n",
    " qualitative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6e10c8fa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.332318Z",
     "iopub.status.busy": "2024-06-04T23:19:10.332231Z",
     "iopub.status.idle": "2024-06-04T23:19:10.336148Z",
     "shell.execute_reply": "2024-06-04T23:19:10.335919Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year</th>\n",
       "      <th>Lag1</th>\n",
       "      <th>Lag2</th>\n",
       "      <th>Lag3</th>\n",
       "      <th>Lag4</th>\n",
       "      <th>Lag5</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Today</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Year</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.029700</td>\n",
       "      <td>0.030596</td>\n",
       "      <td>0.033195</td>\n",
       "      <td>0.035689</td>\n",
       "      <td>0.029788</td>\n",
       "      <td>0.539006</td>\n",
       "      <td>0.030095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lag1</th>\n",
       "      <td>0.029700</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.026294</td>\n",
       "      <td>-0.010803</td>\n",
       "      <td>-0.002986</td>\n",
       "      <td>-0.005675</td>\n",
       "      <td>0.040910</td>\n",
       "      <td>-0.026155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lag2</th>\n",
       "      <td>0.030596</td>\n",
       "      <td>-0.026294</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.025897</td>\n",
       "      <td>-0.010854</td>\n",
       "      <td>-0.003558</td>\n",
       "      <td>-0.043383</td>\n",
       "      <td>-0.010250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lag3</th>\n",
       "      <td>0.033195</td>\n",
       "      <td>-0.010803</td>\n",
       "      <td>-0.025897</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.024051</td>\n",
       "      <td>-0.018808</td>\n",
       "      <td>-0.041824</td>\n",
       "      <td>-0.002448</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lag4</th>\n",
       "      <td>0.035689</td>\n",
       "      <td>-0.002986</td>\n",
       "      <td>-0.010854</td>\n",
       "      <td>-0.024051</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.027084</td>\n",
       "      <td>-0.048414</td>\n",
       "      <td>-0.006900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lag5</th>\n",
       "      <td>0.029788</td>\n",
       "      <td>-0.005675</td>\n",
       "      <td>-0.003558</td>\n",
       "      <td>-0.018808</td>\n",
       "      <td>-0.027084</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.022002</td>\n",
       "      <td>-0.034860</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Volume</th>\n",
       "      <td>0.539006</td>\n",
       "      <td>0.040910</td>\n",
       "      <td>-0.043383</td>\n",
       "      <td>-0.041824</td>\n",
       "      <td>-0.048414</td>\n",
       "      <td>-0.022002</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.014592</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Today</th>\n",
       "      <td>0.030095</td>\n",
       "      <td>-0.026155</td>\n",
       "      <td>-0.010250</td>\n",
       "      <td>-0.002448</td>\n",
       "      <td>-0.006900</td>\n",
       "      <td>-0.034860</td>\n",
       "      <td>0.014592</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Year      Lag1      Lag2      Lag3      Lag4      Lag5    Volume  \\\n",
       "Year    1.000000  0.029700  0.030596  0.033195  0.035689  0.029788  0.539006   \n",
       "Lag1    0.029700  1.000000 -0.026294 -0.010803 -0.002986 -0.005675  0.040910   \n",
       "Lag2    0.030596 -0.026294  1.000000 -0.025897 -0.010854 -0.003558 -0.043383   \n",
       "Lag3    0.033195 -0.010803 -0.025897  1.000000 -0.024051 -0.018808 -0.041824   \n",
       "Lag4    0.035689 -0.002986 -0.010854 -0.024051  1.000000 -0.027084 -0.048414   \n",
       "Lag5    0.029788 -0.005675 -0.003558 -0.018808 -0.027084  1.000000 -0.022002   \n",
       "Volume  0.539006  0.040910 -0.043383 -0.041824 -0.048414 -0.022002  1.000000   \n",
       "Today   0.030095 -0.026155 -0.010250 -0.002448 -0.006900 -0.034860  0.014592   \n",
       "\n",
       "           Today  \n",
       "Year    0.030095  \n",
       "Lag1   -0.026155  \n",
       "Lag2   -0.010250  \n",
       "Lag3   -0.002448  \n",
       "Lag4   -0.006900  \n",
       "Lag5   -0.034860  \n",
       "Volume  0.014592  \n",
       "Today   1.000000  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Smarket.corr(numeric_only=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6659038",
   "metadata": {},
   "source": [
    "As one would expect, the correlations between the lagged return  variables and\n",
    "today’s return are close to zero.  The only substantial correlation is between  `Year`  and\n",
    " `Volume`. By plotting the data we see that  `Volume`\n",
    "is increasing over time. In other words, the average number of shares traded\n",
    "daily increased from 2001 to 2005.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c30034ac",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.337474Z",
     "iopub.status.busy": "2024-06-04T23:19:10.337394Z",
     "iopub.status.idle": "2024-06-04T23:19:10.430280Z",
     "shell.execute_reply": "2024-06-04T23:19:10.429786Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 49796 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "Smarket.plot(y='Volume');\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79919219",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "Next, we will fit a logistic regression model in order to predict\n",
    " `Direction`  using  `Lag1`  through  `Lag5`  and\n",
    " `Volume`. The `sm.GLM()`  function fits *generalized linear models*, a class of\n",
    "models that includes logistic regression.  Alternatively,\n",
    "the function `sm.Logit()` fits a logistic regression\n",
    "model directly. The syntax of\n",
    "`sm.GLM()` is similar to that of `sm.OLS()`, except\n",
    "that we must pass in the argument `family=sm.families.Binomial()`\n",
    "in order to tell `statsmodels` to run a logistic regression rather than some other\n",
    "type of generalized linear model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "61a82664",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.434291Z",
     "iopub.status.busy": "2024-06-04T23:19:10.433853Z",
     "iopub.status.idle": "2024-06-04T23:19:10.501135Z",
     "shell.execute_reply": "2024-06-04T23:19:10.483890Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>z</th>\n",
       "      <th>P&gt;|z|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>-0.1260</td>\n",
       "      <td>0.241</td>\n",
       "      <td>-0.523</td>\n",
       "      <td>0.601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lag1</th>\n",
       "      <td>-0.0731</td>\n",
       "      <td>0.050</td>\n",
       "      <td>-1.457</td>\n",
       "      <td>0.145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lag2</th>\n",
       "      <td>-0.0423</td>\n",
       "      <td>0.050</td>\n",
       "      <td>-0.845</td>\n",
       "      <td>0.398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lag3</th>\n",
       "      <td>0.0111</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.222</td>\n",
       "      <td>0.824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lag4</th>\n",
       "      <td>0.0094</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.187</td>\n",
       "      <td>0.851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lag5</th>\n",
       "      <td>0.0103</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.208</td>\n",
       "      <td>0.835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Volume</th>\n",
       "      <td>0.1354</td>\n",
       "      <td>0.158</td>\n",
       "      <td>0.855</td>\n",
       "      <td>0.392</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             coef  std err      z  P>|z|\n",
       "intercept -0.1260    0.241 -0.523  0.601\n",
       "Lag1      -0.0731    0.050 -1.457  0.145\n",
       "Lag2      -0.0423    0.050 -0.845  0.398\n",
       "Lag3       0.0111    0.050  0.222  0.824\n",
       "Lag4       0.0094    0.050  0.187  0.851\n",
       "Lag5       0.0103    0.050  0.208  0.835\n",
       "Volume     0.1354    0.158  0.855  0.392"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "allvars = Smarket.columns.drop(['Today', 'Direction', 'Year'])\n",
    "design = MS(allvars)\n",
    "X = design.fit_transform(Smarket)\n",
    "y = Smarket.Direction == 'Up'\n",
    "glm = sm.GLM(y,\n",
    "             X,\n",
    "             family=sm.families.Binomial())\n",
    "results = glm.fit()\n",
    "summarize(results)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee972938",
   "metadata": {},
   "source": [
    "The smallest *p*-value here is associated with  `Lag1`. The\n",
    "negative coefficient for this predictor suggests that if the market\n",
    "had a positive return yesterday, then it is less likely to go up\n",
    "today. However, at a value of 0.15, the *p*-value is still\n",
    "relatively large, and so there is no clear evidence of a real\n",
    "association between  `Lag1`  and  `Direction`.\n",
    "\n",
    "We use the `params`  attribute of `results`\n",
    "in order to access just the\n",
    "coefficients for this fitted model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d09a55d9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.504964Z",
     "iopub.status.busy": "2024-06-04T23:19:10.504628Z",
     "iopub.status.idle": "2024-06-04T23:19:10.523026Z",
     "shell.execute_reply": "2024-06-04T23:19:10.521952Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "intercept   -0.126000\n",
       "Lag1        -0.073074\n",
       "Lag2        -0.042301\n",
       "Lag3         0.011085\n",
       "Lag4         0.009359\n",
       "Lag5         0.010313\n",
       "Volume       0.135441\n",
       "dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.params\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c886505",
   "metadata": {},
   "source": [
    "Likewise we can use the\n",
    "`pvalues`  attribute to access the *p*-values for the coefficients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a14e688f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.532342Z",
     "iopub.status.busy": "2024-06-04T23:19:10.530799Z",
     "iopub.status.idle": "2024-06-04T23:19:10.538953Z",
     "shell.execute_reply": "2024-06-04T23:19:10.538379Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "intercept    0.600700\n",
       "Lag1         0.145232\n",
       "Lag2         0.398352\n",
       "Lag3         0.824334\n",
       "Lag4         0.851445\n",
       "Lag5         0.834998\n",
       "Volume       0.392404\n",
       "dtype: float64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.pvalues\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd01f173",
   "metadata": {},
   "source": [
    "The `predict()`  method of `results` can be used to predict the\n",
    "probability that the market will go up, given values of the\n",
    "predictors. This method returns predictions\n",
    "on the probability scale. If no data set is supplied to the `predict()`\n",
    "function, then the probabilities are computed for the training data\n",
    "that was used to fit the logistic regression model.\n",
    "As with linear regression, one can pass an optional `exog` argument consistent\n",
    "with a design matrix if desired. Here we have\n",
    "printed only the first ten probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "2d1d337a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.545854Z",
     "iopub.status.busy": "2024-06-04T23:19:10.545607Z",
     "iopub.status.idle": "2024-06-04T23:19:10.550331Z",
     "shell.execute_reply": "2024-06-04T23:19:10.549806Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.50708413, 0.48146788, 0.48113883, 0.51522236, 0.51078116,\n",
       "       0.50695646, 0.49265087, 0.50922916, 0.51761353, 0.48883778])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probs = results.predict()\n",
    "probs[:10]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c464dc7d",
   "metadata": {},
   "source": [
    "In order to make a prediction as to whether the market will go up or\n",
    "down on a particular day, we must convert these predicted\n",
    "probabilities into class labels,  `Up`  or  `Down`.  The\n",
    "following two commands create a vector of class predictions based on\n",
    "whether the predicted probability of a market increase is greater than\n",
    "or less than 0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "db97e20c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.553209Z",
     "iopub.status.busy": "2024-06-04T23:19:10.553027Z",
     "iopub.status.idle": "2024-06-04T23:19:10.555510Z",
     "shell.execute_reply": "2024-06-04T23:19:10.555019Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "labels = np.array(['Down']*1250)\n",
    "labels[probs>0.5] = \"Up\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2079aa26",
   "metadata": {},
   "source": [
    "The `confusion_table()`\n",
    "function from the `ISLP` package summarizes these predictions, showing   how\n",
    "many observations were correctly or incorrectly classified. Our function, which is adapted from a similar function\n",
    "in the module `sklearn.metrics`,  transposes the resulting\n",
    "matrix and includes row and column labels.\n",
    "The `confusion_table()` function takes as first argument the\n",
    "predicted labels, and second argument the true labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5173815c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.558137Z",
     "iopub.status.busy": "2024-06-04T23:19:10.557755Z",
     "iopub.status.idle": "2024-06-04T23:19:10.568812Z",
     "shell.execute_reply": "2024-06-04T23:19:10.568379Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Truth</th>\n",
       "      <th>Down</th>\n",
       "      <th>Up</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predicted</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Down</th>\n",
       "      <td>145</td>\n",
       "      <td>141</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Up</th>\n",
       "      <td>457</td>\n",
       "      <td>507</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Truth      Down   Up\n",
       "Predicted           \n",
       "Down        145  141\n",
       "Up          457  507"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_table(labels, Smarket.Direction)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e02ffcf4",
   "metadata": {},
   "source": [
    "The diagonal elements of the confusion matrix indicate correct\n",
    "predictions, while the off-diagonals represent incorrect\n",
    "predictions. Hence our model correctly predicted that the market would\n",
    "go up on 507 days and that it would go down on 145 days, for a\n",
    "total of 507 + 145 = 652 correct predictions. The `np.mean()`\n",
    "function can be used to compute the fraction of days for which the\n",
    "prediction was correct. In this case, logistic regression correctly\n",
    "predicted the movement of the market 52.2% of the time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b49ce3a2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.571227Z",
     "iopub.status.busy": "2024-06-04T23:19:10.571020Z",
     "iopub.status.idle": "2024-06-04T23:19:10.575331Z",
     "shell.execute_reply": "2024-06-04T23:19:10.574762Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.5216, 0.5216)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(507+145)/1250, np.mean(labels == Smarket.Direction)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "547a999d",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad43edf5",
   "metadata": {},
   "source": [
    "At first glance, it appears that the logistic regression model is\n",
    "working a little better than random guessing. However, this result is\n",
    "misleading because we trained and tested the model on the same set of\n",
    "1,250 observations. In other words, $100-52.2=47.8%$ is the\n",
    "*training* error  rate. As we have seen\n",
    "previously, the training error rate is often overly optimistic --- it\n",
    "tends to underestimate the test error rate.  In\n",
    "order to better assess the accuracy of the logistic regression model\n",
    "in this setting, we can fit the model using part of the data, and\n",
    "then examine how well it predicts the *held out* data.  This\n",
    "will yield a more realistic error rate, in the sense that in practice\n",
    "we will be interested in our model’s performance not on the data that\n",
    "we used to fit the model, but rather on days in the future for which\n",
    "the market’s movements are unknown.\n",
    "\n",
    "To implement this strategy, we first create a Boolean vector\n",
    "corresponding to the observations from 2001 through 2004. We  then\n",
    "use this vector to create a held out data set of observations from\n",
    "2005."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "773ab528",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.578285Z",
     "iopub.status.busy": "2024-06-04T23:19:10.578069Z",
     "iopub.status.idle": "2024-06-04T23:19:10.582814Z",
     "shell.execute_reply": "2024-06-04T23:19:10.582379Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(252, 9)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = (Smarket.Year < 2005)\n",
    "Smarket_train = Smarket.loc[train]\n",
    "Smarket_test = Smarket.loc[~train]\n",
    "Smarket_test.shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a416bea",
   "metadata": {},
   "source": [
    "The object `train` is a vector of 1,250 elements, corresponding\n",
    "to the observations in our data set. The elements of the vector that\n",
    "correspond to observations that occurred before 2005 are set to\n",
    "`True`, whereas those that correspond to observations in 2005 are\n",
    "set to `False`.  Hence `train` is a\n",
    "*boolean*   array, since its\n",
    "elements are `True` and `False`.  Boolean arrays can be used\n",
    "to obtain a subset of the rows or columns of a data frame\n",
    "using the `loc` method. For instance,\n",
    "the command `Smarket.loc[train]` would pick out a submatrix of the\n",
    "stock market data set, corresponding only to the dates before 2005,\n",
    "since those are the ones for which the elements of `train` are\n",
    "`True`.  The `~` symbol can be used to negate all of the\n",
    "elements of a Boolean vector. That is, `~train` is a vector\n",
    "similar to `train`, except that the elements that are `True`\n",
    "in `train` get swapped to `False` in `~train`, and vice versa.\n",
    "Therefore, `Smarket.loc[~train]` yields a\n",
    "subset of the rows of the data frame\n",
    "of the stock market data containing only the observations for which\n",
    "`train` is `False`.\n",
    "The output above indicates that there are 252 such\n",
    "observations.\n",
    "\n",
    "We now fit a logistic regression model using only the subset of the\n",
    "observations that correspond to dates before 2005. We then obtain predicted probabilities of the\n",
    "stock market going up for each of the days in our test set --- that is,\n",
    "for the days in 2005."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e8b1b1e8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.585518Z",
     "iopub.status.busy": "2024-06-04T23:19:10.585316Z",
     "iopub.status.idle": "2024-06-04T23:19:10.592938Z",
     "shell.execute_reply": "2024-06-04T23:19:10.592438Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train, X_test = X.loc[train], X.loc[~train]\n",
    "y_train, y_test = y.loc[train], y.loc[~train]\n",
    "glm_train = sm.GLM(y_train,\n",
    "                   X_train,\n",
    "                   family=sm.families.Binomial())\n",
    "results = glm_train.fit()\n",
    "probs = results.predict(exog=X_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "230e2b8e",
   "metadata": {},
   "source": [
    "Notice that we have trained and tested our model on two completely\n",
    "separate data sets: training was performed using only the dates before\n",
    "2005, and testing was performed using only the dates in 2005.\n",
    "\n",
    "Finally, we compare the predictions for 2005 to the\n",
    "actual movements of the market over that time period.\n",
    "We will first store the test and training labels (recall `y_test` is binary)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c1993ea5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.595809Z",
     "iopub.status.busy": "2024-06-04T23:19:10.595620Z",
     "iopub.status.idle": "2024-06-04T23:19:10.599118Z",
     "shell.execute_reply": "2024-06-04T23:19:10.598574Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "D = Smarket.Direction\n",
    "L_train, L_test = D.loc[train], D.loc[~train]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbc2679d",
   "metadata": {},
   "source": [
    "Now we threshold the\n",
    "fitted probability at 50% to form\n",
    "our predicted labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8f909de4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.601671Z",
     "iopub.status.busy": "2024-06-04T23:19:10.601471Z",
     "iopub.status.idle": "2024-06-04T23:19:10.609107Z",
     "shell.execute_reply": "2024-06-04T23:19:10.608569Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Truth</th>\n",
       "      <th>Down</th>\n",
       "      <th>Up</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predicted</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Down</th>\n",
       "      <td>77</td>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Up</th>\n",
       "      <td>34</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Truth      Down  Up\n",
       "Predicted          \n",
       "Down         77  97\n",
       "Up           34  44"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels = np.array(['Down']*252)\n",
    "labels[probs>0.5] = 'Up'\n",
    "confusion_table(labels, L_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8145ecec",
   "metadata": {},
   "source": [
    "The test accuracy is about 48% while the error rate is about 52%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "3c479105",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.611827Z",
     "iopub.status.busy": "2024-06-04T23:19:10.611594Z",
     "iopub.status.idle": "2024-06-04T23:19:10.615795Z",
     "shell.execute_reply": "2024-06-04T23:19:10.615260Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.4801587301587302, 0.5198412698412699)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(labels == L_test), np.mean(labels != L_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b1609dd",
   "metadata": {},
   "source": [
    "The `!=` notation means *not equal to*, and so the last command\n",
    "computes the test set error rate. The results are rather\n",
    "disappointing: the test error rate is 52%, which is worse than\n",
    "random guessing! Of course this result is not all that surprising,\n",
    "given that one would not generally expect to be able to use previous\n",
    "days’ returns to predict future market performance. (After all, if it\n",
    "were possible to do so, then the authors of this book would be out\n",
    "striking it rich rather than writing a statistics textbook.)\n",
    "\n",
    "We recall that the logistic regression model had very underwhelming\n",
    "*p*-values associated with all of the predictors, and that the\n",
    "smallest *p*-value, though not very small, corresponded to\n",
    " `Lag1`. Perhaps by removing the variables that appear not to be\n",
    "helpful in predicting  `Direction`, we can obtain a more\n",
    "effective model. After all, using predictors that have no relationship\n",
    "with the response tends to cause a deterioration in the test error\n",
    "rate (since such predictors cause an increase in variance without a\n",
    "corresponding decrease in bias), and so removing such predictors may\n",
    "in turn yield an improvement.  Below we refit the logistic\n",
    "regression using just  `Lag1`  and  `Lag2`, which seemed to\n",
    "have the highest predictive power in the original logistic regression\n",
    "model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "3f473ec0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.618328Z",
     "iopub.status.busy": "2024-06-04T23:19:10.618134Z",
     "iopub.status.idle": "2024-06-04T23:19:10.657121Z",
     "shell.execute_reply": "2024-06-04T23:19:10.656851Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Truth</th>\n",
       "      <th>Down</th>\n",
       "      <th>Up</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predicted</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Down</th>\n",
       "      <td>35</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Up</th>\n",
       "      <td>76</td>\n",
       "      <td>106</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Truth      Down   Up\n",
       "Predicted           \n",
       "Down         35   35\n",
       "Up           76  106"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = MS(['Lag1', 'Lag2']).fit(Smarket)\n",
    "X = model.transform(Smarket)\n",
    "X_train, X_test = X.loc[train], X.loc[~train]\n",
    "glm_train = sm.GLM(y_train,\n",
    "                   X_train,\n",
    "                   family=sm.families.Binomial())\n",
    "results = glm_train.fit()\n",
    "probs = results.predict(exog=X_test)\n",
    "labels = np.array(['Down']*252)\n",
    "labels[probs>0.5] = 'Up'\n",
    "confusion_table(labels, L_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46cf03c1",
   "metadata": {},
   "source": [
    "Let’s evaluate the overall accuracy as well as the accuracy within the days when\n",
    "logistic regression predicts an increase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b3cd8b84",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.658566Z",
     "iopub.status.busy": "2024-06-04T23:19:10.658490Z",
     "iopub.status.idle": "2024-06-04T23:19:10.660432Z",
     "shell.execute_reply": "2024-06-04T23:19:10.660219Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.5595238095238095, 0.5824175824175825)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(35+106)/252,106/(106+76)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4641aad1",
   "metadata": {},
   "source": [
    "Now the results appear to be a little better: 56% of the daily\n",
    "movements have been correctly predicted. It is worth noting that in\n",
    "this case, a much simpler strategy of predicting that the market will\n",
    "increase every day will also be correct 56% of the time! Hence, in\n",
    "terms of overall error rate, the logistic regression method is no\n",
    "better than the naive approach. However, the confusion matrix\n",
    "shows that on days when logistic regression predicts an increase in\n",
    "the market, it has a 58% accuracy rate. This suggests a possible\n",
    "trading strategy of buying on days when the model predicts an\n",
    "increasing market, and avoiding trades on days when a decrease is\n",
    "predicted. Of course one would need to investigate more carefully\n",
    "whether this small improvement was real or just due to random chance.\n",
    "\n",
    "Suppose that we want to predict the returns associated with particular\n",
    "values of  `Lag1`  and  `Lag2`. In particular, we want to\n",
    "predict  `Direction`  on a day when  `Lag1`  and\n",
    " `Lag2`  equal $1.2$ and $1.1$, respectively, and on a day when they\n",
    "equal $1.5$ and $-0.8$.  We do this using the `predict()`\n",
    "function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d15e7495",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.661646Z",
     "iopub.status.busy": "2024-06-04T23:19:10.661582Z",
     "iopub.status.idle": "2024-06-04T23:19:10.664591Z",
     "shell.execute_reply": "2024-06-04T23:19:10.664373Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.479146\n",
       "1    0.496094\n",
       "dtype: float64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "newdata = pd.DataFrame({'Lag1':[1.2, 1.5],\n",
    "                        'Lag2':[1.1, -0.8]});\n",
    "newX = model.transform(newdata)\n",
    "results.predict(newX)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "595d2fa9",
   "metadata": {},
   "source": [
    "## Linear Discriminant Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d31037d",
   "metadata": {},
   "source": [
    "We begin by performing LDA on the  `Smarket`  data, using the function\n",
    "`LinearDiscriminantAnalysis()`, which we have abbreviated `LDA()`. We \n",
    "fit the model using only the observations before 2005."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "586b8bc1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.665995Z",
     "iopub.status.busy": "2024-06-04T23:19:10.665887Z",
     "iopub.status.idle": "2024-06-04T23:19:10.667484Z",
     "shell.execute_reply": "2024-06-04T23:19:10.667242Z"
    }
   },
   "outputs": [],
   "source": [
    "lda = LDA(store_covariance=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43672635",
   "metadata": {},
   "source": [
    "Since the `LDA` estimator automatically \n",
    "adds an intercept, we should remove the column corresponding to the\n",
    "intercept in both `X_train` and `X_test`. We can also directly\n",
    "use the labels rather than the Boolean vectors `y_train`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "18fa8ae5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.668696Z",
     "iopub.status.busy": "2024-06-04T23:19:10.668623Z",
     "iopub.status.idle": "2024-06-04T23:19:10.673662Z",
     "shell.execute_reply": "2024-06-04T23:19:10.673442Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LinearDiscriminantAnalysis(store_covariance=True)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;LinearDiscriminantAnalysis<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html\">?<span>Documentation for LinearDiscriminantAnalysis</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>LinearDiscriminantAnalysis(store_covariance=True)</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "LinearDiscriminantAnalysis(store_covariance=True)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train, X_test = [M.drop(columns=['intercept'])\n",
    "                   for M in [X_train, X_test]]\n",
    "lda.fit(X_train, L_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8254b5d",
   "metadata": {},
   "source": [
    "Here we have used the list comprehensions introduced\n",
    "in Section~\\ref{Ch3-linreg-lab:multivariate-goodness-of-fit}. Looking at our first line above, we see that the right-hand side is a list\n",
    "of length two. This is because the code `for M in [X_train, X_test]` iterates over a list\n",
    "of length two. While here we loop over a list,\n",
    "the list comprehension method works when looping over any iterable object.\n",
    "We then apply the `drop()` method to each element in the iteration, collecting\n",
    "the result in a list. The left-hand side tells `Python` to unpack this list\n",
    "of length two, assigning its elements to the variables `X_train` and `X_test`. Of course,\n",
    "this overwrites the previous values of `X_train` and `X_test`.\n",
    "\n",
    "Having fit the model, we can extract the means in the two classes with the `means_` attribute. These are the average of each predictor within each class, and\n",
    "are used by LDA as estimates of $\\mu_k$.  These suggest that there is\n",
    "a tendency for the previous 2 days’ returns to be negative on days\n",
    "when the market increases, and a tendency for the previous days’\n",
    "returns to be positive on days when the market declines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "4c9a8391",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.674879Z",
     "iopub.status.busy": "2024-06-04T23:19:10.674804Z",
     "iopub.status.idle": "2024-06-04T23:19:10.676917Z",
     "shell.execute_reply": "2024-06-04T23:19:10.676691Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.04279022,  0.03389409],\n",
       "       [-0.03954635, -0.03132544]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.means_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d21cb4e",
   "metadata": {},
   "source": [
    "The estimated prior probabilities are stored in the `priors_` attribute.\n",
    "The package `sklearn` typically uses this trailing `_` to denote\n",
    "a quantity estimated when using the `fit()` method. We can be sure of which\n",
    "entry corresponds to which label by looking at the `classes_` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "0b774571",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.678089Z",
     "iopub.status.busy": "2024-06-04T23:19:10.678014Z",
     "iopub.status.idle": "2024-06-04T23:19:10.679873Z",
     "shell.execute_reply": "2024-06-04T23:19:10.679660Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Down', 'Up'], dtype='<U4')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.classes_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a959865",
   "metadata": {},
   "source": [
    "The LDA output indicates that $\\hat\\pi_{Down}=0.492$ and\n",
    "$\\hat\\pi_{Up}=0.508$.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "148f227f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.680993Z",
     "iopub.status.busy": "2024-06-04T23:19:10.680932Z",
     "iopub.status.idle": "2024-06-04T23:19:10.682749Z",
     "shell.execute_reply": "2024-06-04T23:19:10.682530Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.49198397, 0.50801603])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.priors_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2351d5f",
   "metadata": {},
   "source": [
    "The linear discriminant vectors can be found in the `scalings_` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "73068c40",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.683928Z",
     "iopub.status.busy": "2024-06-04T23:19:10.683852Z",
     "iopub.status.idle": "2024-06-04T23:19:10.685758Z",
     "shell.execute_reply": "2024-06-04T23:19:10.685565Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.64201904],\n",
       "       [-0.51352928]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.scalings_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0a4abaf",
   "metadata": {},
   "source": [
    "These values provide the linear combination of `Lag1`  and `Lag2`  that are used to form the LDA decision rule. In other words, these are the multipliers of the elements of $X=x$ in (\\ref{Ch4:bayes.multi}).\n",
    "  If $-0.64\\times `Lag1`  - 0.51 \\times `Lag2` $ is large, then the LDA classifier will predict a market increase, and if it is small, then the LDA classifier will predict a market decline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "a8d52279",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.686970Z",
     "iopub.status.busy": "2024-06-04T23:19:10.686905Z",
     "iopub.status.idle": "2024-06-04T23:19:10.688834Z",
     "shell.execute_reply": "2024-06-04T23:19:10.688607Z"
    }
   },
   "outputs": [],
   "source": [
    "lda_pred = lda.predict(X_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "148cc469",
   "metadata": {},
   "source": [
    "As we observed in our comparison of classification methods\n",
    " (Section~\\ref{Ch4:comparison.sec}),  the LDA and logistic\n",
    "regression predictions are almost identical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "b83cbac2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.690007Z",
     "iopub.status.busy": "2024-06-04T23:19:10.689936Z",
     "iopub.status.idle": "2024-06-04T23:19:10.693543Z",
     "shell.execute_reply": "2024-06-04T23:19:10.693330Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Truth</th>\n",
       "      <th>Down</th>\n",
       "      <th>Up</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predicted</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Down</th>\n",
       "      <td>35</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Up</th>\n",
       "      <td>76</td>\n",
       "      <td>106</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Truth      Down   Up\n",
       "Predicted           \n",
       "Down         35   35\n",
       "Up           76  106"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_table(lda_pred, L_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2261b398",
   "metadata": {},
   "source": [
    "We can also estimate the\n",
    "probability of each class for\n",
    "each point in a training set. Applying a 50% threshold to the posterior probabilities of\n",
    "being in class one allows us to\n",
    "recreate the predictions contained in `lda_pred`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "496f213c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.694765Z",
     "iopub.status.busy": "2024-06-04T23:19:10.694696Z",
     "iopub.status.idle": "2024-06-04T23:19:10.697074Z",
     "shell.execute_reply": "2024-06-04T23:19:10.696863Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_prob = lda.predict_proba(X_test)\n",
    "np.all(\n",
    "       np.where(lda_prob[:,1] >= 0.5, 'Up','Down') == lda_pred\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56d0ad8f",
   "metadata": {},
   "source": [
    "Above, we used the `np.where()`  function that\n",
    "creates an array with value `'Up'` for indices where\n",
    "the second column of `lda_prob` (the estimated\n",
    "posterior probability of `'Up'`) is greater than 0.5.\n",
    "For problems with more than two classes the labels are chosen as the class whose posterior probability is highest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7a306b42",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.698382Z",
     "iopub.status.busy": "2024-06-04T23:19:10.698315Z",
     "iopub.status.idle": "2024-06-04T23:19:10.700428Z",
     "shell.execute_reply": "2024-06-04T23:19:10.700221Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.all(\n",
    "       [lda.classes_[i] for i in np.argmax(lda_prob, 1)] == lda_pred\n",
    "       )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01fe47ab",
   "metadata": {},
   "source": [
    "If we wanted to use a posterior probability threshold other than\n",
    "50% in order to make predictions, then we could easily do so. For\n",
    "instance, suppose that we wish to predict a market decrease only if we\n",
    "are very certain that the market will indeed decrease on that\n",
    "day --- say, if the posterior probability is at least 90%.\n",
    "We know that the first column of `lda_prob` corresponds to the\n",
    "label `Down` after having checked the `classes_` attribute, hence we use\n",
    "the column index 0 rather than 1 as we did above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f2d7878b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.701614Z",
     "iopub.status.busy": "2024-06-04T23:19:10.701548Z",
     "iopub.status.idle": "2024-06-04T23:19:10.703307Z",
     "shell.execute_reply": "2024-06-04T23:19:10.703099Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(lda_prob[:,0] > 0.9)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5c3e881",
   "metadata": {},
   "source": [
    "No days in 2005 meet that threshold! In fact, the greatest posterior\n",
    "probability of decrease in all of 2005 was 52.02%.\n",
    "\n",
    "The LDA classifier above is the first classifier from the\n",
    "`sklearn` library. We will use several other objects\n",
    "from this library. The objects\n",
    "follow a common structure that simplifies tasks such as cross-validation,\n",
    "which we will see in Chapter~\\ref{Ch5:resample}. Specifically,\n",
    "the methods first create a generic classifier without\n",
    "referring to any data. This classifier is then fit\n",
    "to data with the `fit()`  method and predictions are\n",
    "always produced with the `predict()` method. This pattern\n",
    "of first instantiating the classifier, followed by fitting it, and\n",
    "then producing predictions is an explicit design choice of `sklearn`. This uniformity\n",
    "makes it possible to cleanly copy the classifier so that it can be fit\n",
    "on different data; e.g. different training sets arising in cross-validation.\n",
    "This standard pattern also allows for a predictable formation of workflows.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbeab8f9",
   "metadata": {},
   "source": [
    "## Quadratic Discriminant Analysis\n",
    "We will now fit a QDA model to the  `Smarket`  data. QDA is\n",
    "implemented via\n",
    "`QuadraticDiscriminantAnalysis()`\n",
    "in the `sklearn` package, which we abbreviate to `QDA()`.\n",
    "The syntax is very similar to `LDA()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "6fc87c48",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.704453Z",
     "iopub.status.busy": "2024-06-04T23:19:10.704385Z",
     "iopub.status.idle": "2024-06-04T23:19:10.707626Z",
     "shell.execute_reply": "2024-06-04T23:19:10.707394Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-2 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-2 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-2 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-2 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-2 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-2 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>QuadraticDiscriminantAnalysis(store_covariance=True)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;QuadraticDiscriminantAnalysis<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.discriminant_analysis.QuadraticDiscriminantAnalysis.html\">?<span>Documentation for QuadraticDiscriminantAnalysis</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>QuadraticDiscriminantAnalysis(store_covariance=True)</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "QuadraticDiscriminantAnalysis(store_covariance=True)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qda = QDA(store_covariance=True)\n",
    "qda.fit(X_train, L_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a0ca885",
   "metadata": {},
   "source": [
    "The `QDA()` function will again compute `means_` and `priors_`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "92f4f928",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.708905Z",
     "iopub.status.busy": "2024-06-04T23:19:10.708836Z",
     "iopub.status.idle": "2024-06-04T23:19:10.710715Z",
     "shell.execute_reply": "2024-06-04T23:19:10.710514Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[ 0.04279022,  0.03389409],\n",
       "        [-0.03954635, -0.03132544]]),\n",
       " array([0.49198397, 0.50801603]))"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qda.means_, qda.priors_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c431c86f",
   "metadata": {},
   "source": [
    "The `QDA()` classifier will estimate one covariance per class. Here is the\n",
    "estimated covariance in the first class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "d016f22c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.711933Z",
     "iopub.status.busy": "2024-06-04T23:19:10.711857Z",
     "iopub.status.idle": "2024-06-04T23:19:10.713787Z",
     "shell.execute_reply": "2024-06-04T23:19:10.713572Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.50662277, -0.03924806],\n",
       "       [-0.03924806,  1.53559498]])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qda.covariance_[0]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9255f459",
   "metadata": {},
   "source": [
    "The output contains the group means. But it does not contain the\n",
    "coefficients of the linear discriminants, because the QDA classifier\n",
    "involves a quadratic, rather than a linear, function of the\n",
    "predictors. The `predict()`  function works in exactly the\n",
    "same fashion as for LDA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "0c8fa11a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.714998Z",
     "iopub.status.busy": "2024-06-04T23:19:10.714933Z",
     "iopub.status.idle": "2024-06-04T23:19:10.718857Z",
     "shell.execute_reply": "2024-06-04T23:19:10.718646Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Truth</th>\n",
       "      <th>Down</th>\n",
       "      <th>Up</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predicted</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Down</th>\n",
       "      <td>30</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Up</th>\n",
       "      <td>81</td>\n",
       "      <td>121</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Truth      Down   Up\n",
       "Predicted           \n",
       "Down         30   20\n",
       "Up           81  121"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qda_pred = qda.predict(X_test)\n",
    "confusion_table(qda_pred, L_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d80294b",
   "metadata": {},
   "source": [
    "Interestingly, the QDA predictions are accurate almost 60% of the\n",
    "time, even though the 2005 data was not used to fit the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "b010de50",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.720069Z",
     "iopub.status.busy": "2024-06-04T23:19:10.720000Z",
     "iopub.status.idle": "2024-06-04T23:19:10.721978Z",
     "shell.execute_reply": "2024-06-04T23:19:10.721773Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5992063492063492"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(qda_pred == L_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b57525d1",
   "metadata": {},
   "source": [
    "This level of accuracy is quite impressive for stock market data, which is\n",
    "known to be quite hard to model accurately.  This suggests that the\n",
    "quadratic form assumed by QDA may capture the true relationship more\n",
    "accurately than the linear forms assumed by LDA and logistic\n",
    "regression.  However, we recommend evaluating this method’s\n",
    "performance on a larger test set before betting that this approach\n",
    "will consistently beat the market!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "152e9b52",
   "metadata": {},
   "source": [
    "## Naive Bayes\n",
    "Next, we fit a naive Bayes model to the `Smarket` data. The syntax is\n",
    "similar to that of `LDA()` and `QDA()`. By\n",
    "default, this implementation `GaussianNB()` of the naive Bayes classifier models each\n",
    "quantitative feature using a Gaussian distribution. However, a kernel\n",
    "density method can also be used to estimate the distributions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "78cac089",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.723135Z",
     "iopub.status.busy": "2024-06-04T23:19:10.723068Z",
     "iopub.status.idle": "2024-06-04T23:19:10.726306Z",
     "shell.execute_reply": "2024-06-04T23:19:10.726108Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-3 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-3 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-3 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-3 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-3 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-3 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-3 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-3 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-3 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-3 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-3 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GaussianNB()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" checked><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;GaussianNB<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.naive_bayes.GaussianNB.html\">?<span>Documentation for GaussianNB</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>GaussianNB()</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "GaussianNB()"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NB = GaussianNB()\n",
    "NB.fit(X_train, L_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0da48d75",
   "metadata": {},
   "source": [
    "The classes are stored as `classes_`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "acce3488",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.727548Z",
     "iopub.status.busy": "2024-06-04T23:19:10.727484Z",
     "iopub.status.idle": "2024-06-04T23:19:10.729522Z",
     "shell.execute_reply": "2024-06-04T23:19:10.729308Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Down', 'Up'], dtype='<U4')"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NB.classes_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6df1a20e",
   "metadata": {},
   "source": [
    "The class prior probabilities are stored in the `class_prior_` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "c176be69",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.730707Z",
     "iopub.status.busy": "2024-06-04T23:19:10.730633Z",
     "iopub.status.idle": "2024-06-04T23:19:10.732656Z",
     "shell.execute_reply": "2024-06-04T23:19:10.732431Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.49198397, 0.50801603])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NB.class_prior_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "276b1efe",
   "metadata": {},
   "source": [
    "The parameters of the features can be found in the `theta_` and `var_` attributes. The number of rows\n",
    "is equal to the number of classes, while the number of columns is equal to the number of features.\n",
    "We see below that the mean for feature `Lag1` in the `Down` class is 0.043."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "bb1b5bed",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.733942Z",
     "iopub.status.busy": "2024-06-04T23:19:10.733876Z",
     "iopub.status.idle": "2024-06-04T23:19:10.735757Z",
     "shell.execute_reply": "2024-06-04T23:19:10.735550Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.04279022,  0.03389409],\n",
       "       [-0.03954635, -0.03132544]])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NB.theta_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e679eb16",
   "metadata": {},
   "source": [
    "Its variance is 1.503."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "158e9f9f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.736918Z",
     "iopub.status.busy": "2024-06-04T23:19:10.736848Z",
     "iopub.status.idle": "2024-06-04T23:19:10.738666Z",
     "shell.execute_reply": "2024-06-04T23:19:10.738459Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.50355429, 1.53246749],\n",
       "       [1.51401364, 1.48732877]])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NB.var_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ba86b52",
   "metadata": {},
   "source": [
    "How do we know the names of these attributes? We use `NB?` (or `?NB`).\n",
    "\n",
    "We can easily verify the mean computation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "eed68a42",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.739817Z",
     "iopub.status.busy": "2024-06-04T23:19:10.739748Z",
     "iopub.status.idle": "2024-06-04T23:19:10.742249Z",
     "shell.execute_reply": "2024-06-04T23:19:10.742035Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Lag1    0.042790\n",
       "Lag2    0.033894\n",
       "dtype: float64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[L_train == 'Down'].mean()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5b73a4d",
   "metadata": {},
   "source": [
    "Similarly for the variance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "3e4e63f1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.743382Z",
     "iopub.status.busy": "2024-06-04T23:19:10.743316Z",
     "iopub.status.idle": "2024-06-04T23:19:10.745780Z",
     "shell.execute_reply": "2024-06-04T23:19:10.745564Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Lag1    1.503554\n",
       "Lag2    1.532467\n",
       "dtype: float64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[L_train == 'Down'].var(ddof=0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f63c729a",
   "metadata": {},
   "source": [
    "Since `NB()` is a classifier in the `sklearn` library, making predictions\n",
    "uses the same syntax as for `LDA()` and `QDA()` above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "560b277f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.746988Z",
     "iopub.status.busy": "2024-06-04T23:19:10.746915Z",
     "iopub.status.idle": "2024-06-04T23:19:10.750768Z",
     "shell.execute_reply": "2024-06-04T23:19:10.750556Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Truth</th>\n",
       "      <th>Down</th>\n",
       "      <th>Up</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predicted</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Down</th>\n",
       "      <td>29</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Up</th>\n",
       "      <td>82</td>\n",
       "      <td>121</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Truth      Down   Up\n",
       "Predicted           \n",
       "Down         29   20\n",
       "Up           82  121"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb_labels = NB.predict(X_test)\n",
    "confusion_table(nb_labels, L_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a16e034c",
   "metadata": {},
   "source": [
    "Naive Bayes performs well on these data, with accurate predictions over 59% of the time. This is slightly worse than QDA, but much better than LDA.\n",
    "\n",
    "As for `LDA`, the `predict_proba()` method estimates the probability that each observation belongs to a particular class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "1efe1d6a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.751934Z",
     "iopub.status.busy": "2024-06-04T23:19:10.751870Z",
     "iopub.status.idle": "2024-06-04T23:19:10.754306Z",
     "shell.execute_reply": "2024-06-04T23:19:10.754090Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.4873288 , 0.5126712 ],\n",
       "       [0.47623584, 0.52376416],\n",
       "       [0.46529531, 0.53470469],\n",
       "       [0.47484469, 0.52515531],\n",
       "       [0.49020587, 0.50979413]])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NB.predict_proba(X_test)[:5]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed4f37e8",
   "metadata": {},
   "source": [
    "## K-Nearest Neighbors\n",
    "We will now perform KNN using the `KNeighborsClassifier()` function. This function works similarly\n",
    "to the other model-fitting functions that we have\n",
    "encountered thus far.\n",
    "\n",
    "As is the\n",
    "case for LDA and QDA, we fit the classifier\n",
    "using the `fit` method. New\n",
    "predictions are formed using the `predict` method\n",
    "of the object returned by `fit()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "b0b8ae27",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.755478Z",
     "iopub.status.busy": "2024-06-04T23:19:10.755409Z",
     "iopub.status.idle": "2024-06-04T23:19:10.763883Z",
     "shell.execute_reply": "2024-06-04T23:19:10.763665Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Truth</th>\n",
       "      <th>Down</th>\n",
       "      <th>Up</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predicted</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Down</th>\n",
       "      <td>43</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Up</th>\n",
       "      <td>68</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Truth      Down  Up\n",
       "Predicted          \n",
       "Down         43  58\n",
       "Up           68  83"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn1 = KNeighborsClassifier(n_neighbors=1)\n",
    "X_train, X_test = [np.asarray(X) for X in [X_train, X_test]]\n",
    "knn1.fit(X_train, L_train)\n",
    "knn1_pred = knn1.predict(X_test)\n",
    "confusion_table(knn1_pred, L_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84c89555",
   "metadata": {},
   "source": [
    "The results using $K=1$ are not very good, since only $50%$ of the\n",
    "observations are correctly predicted. Of course, it may be that $K=1$\n",
    "results in an overly-flexible fit to the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "7c9bdd8e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.765113Z",
     "iopub.status.busy": "2024-06-04T23:19:10.765038Z",
     "iopub.status.idle": "2024-06-04T23:19:10.767093Z",
     "shell.execute_reply": "2024-06-04T23:19:10.766831Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.5, 0.5)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(83+43)/252, np.mean(knn1_pred == L_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96e335c6",
   "metadata": {},
   "source": [
    "We repeat the\n",
    "analysis below using $K=3$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "a1750e64",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.768344Z",
     "iopub.status.busy": "2024-06-04T23:19:10.768258Z",
     "iopub.status.idle": "2024-06-04T23:19:10.774690Z",
     "shell.execute_reply": "2024-06-04T23:19:10.774467Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5317460317460317"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn3 = KNeighborsClassifier(n_neighbors=3)\n",
    "knn3_pred = knn3.fit(X_train, L_train).predict(X_test)\n",
    "np.mean(knn3_pred == L_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecbad54a",
   "metadata": {},
   "source": [
    "The results have improved slightly. But increasing *K* further\n",
    "provides no further improvements. It appears that for these data, and this train/test split,\n",
    "QDA gives the best results of the methods that we have examined so\n",
    "far."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3fe0e0d",
   "metadata": {},
   "source": [
    "KNN does not perform well on the `Smarket`  data, but it often does provide impressive results. As an example we will apply the KNN approach to the `Caravan`  data set, which is part of the `ISLP` library.  This data set includes 85\n",
    "predictors that measure demographic characteristics for 5,822\n",
    "individuals. The response variable is  `Purchase`, which\n",
    "indicates whether or not a given individual purchases a caravan\n",
    "insurance policy. In this data set, only 6% of people purchased\n",
    "caravan insurance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "2b179be8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.776042Z",
     "iopub.status.busy": "2024-06-04T23:19:10.775972Z",
     "iopub.status.idle": "2024-06-04T23:19:10.790666Z",
     "shell.execute_reply": "2024-06-04T23:19:10.790430Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Purchase\n",
       "No     5474\n",
       "Yes     348\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Caravan = load_data('Caravan')\n",
    "Purchase = Caravan.Purchase\n",
    "Purchase.value_counts()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99805400",
   "metadata": {},
   "source": [
    "The method `value_counts()` takes a `pd.Series` or `pd.DataFrame` and returns\n",
    "a `pd.Series` with the corresponding counts\n",
    "for each unique element. In this case `Purchase` has only `Yes` and `No` values\n",
    "and the method returns how many values of each there are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "e9de7237",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.791954Z",
     "iopub.status.busy": "2024-06-04T23:19:10.791856Z",
     "iopub.status.idle": "2024-06-04T23:19:10.793719Z",
     "shell.execute_reply": "2024-06-04T23:19:10.793502Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.05977327378907592"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "348 / 5822\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3fdbe45",
   "metadata": {},
   "source": [
    "Our features will include all columns except `Purchase`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "f81dcb72",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.794971Z",
     "iopub.status.busy": "2024-06-04T23:19:10.794900Z",
     "iopub.status.idle": "2024-06-04T23:19:10.796754Z",
     "shell.execute_reply": "2024-06-04T23:19:10.796532Z"
    }
   },
   "outputs": [],
   "source": [
    "feature_df = Caravan.drop(columns=['Purchase'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f92eadb",
   "metadata": {},
   "source": [
    "Because the KNN classifier predicts the class of a given test\n",
    "observation by identifying the observations that are nearest to it,\n",
    "the scale of the variables matters. Any variables that are on a large\n",
    "scale will have a much larger effect on the *distance* between\n",
    "the observations, and hence on the KNN classifier, than variables that\n",
    "are on a small scale. For instance, imagine a data set that contains\n",
    "two variables,  `salary`  and  `age`  (measured in dollars\n",
    "and years, respectively). As far as KNN is concerned, a difference of\n",
    "1,000 USD in salary is enormous compared to a difference of 50 years in\n",
    "age. Consequently,  `salary`  will drive the KNN classification\n",
    "results, and  `age`  will have almost no effect. This is contrary\n",
    "to our intuition that a salary difference of 1,000 USD is quite small\n",
    "compared to an age difference of 50 years.  Furthermore, the\n",
    "importance of scale to the KNN classifier leads to another issue: if\n",
    "we measured  `salary`  in Japanese yen, or if we measured\n",
    " `age`  in minutes, then we’d get quite different classification\n",
    "results from what we get if these two variables are measured in\n",
    "dollars and years.\n",
    "\n",
    "A good way to handle this problem is to *standardize*  the data so that all variables are\n",
    "given a mean of zero and a standard deviation of one. Then all\n",
    "variables will be on a comparable scale. This is accomplished\n",
    "using\n",
    "the `StandardScaler()`\n",
    "transformation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "a7102e7d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.797953Z",
     "iopub.status.busy": "2024-06-04T23:19:10.797884Z",
     "iopub.status.idle": "2024-06-04T23:19:10.799335Z",
     "shell.execute_reply": "2024-06-04T23:19:10.799104Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "scaler = StandardScaler(with_mean=True,\n",
    "                        with_std=True,\n",
    "                        copy=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaf2eb3d",
   "metadata": {},
   "source": [
    "The argument `with_mean` indicates whether or not\n",
    "we should subtract the mean, while `with_std` indicates\n",
    "whether or not we should scale the columns to have standard\n",
    "deviation of 1 or not. Finally, the argument `copy=True`\n",
    "indicates that we will always copy data, rather than\n",
    "trying to do calculations in place where possible.\n",
    "\n",
    "This transformation can be fit\n",
    "and then applied to arbitrary data. In the first line\n",
    "below, the parameters for the scaling are computed and\n",
    "stored in `scaler`, while the second line actually\n",
    "constructs the standardized set of features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "c2b6c3fa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.800622Z",
     "iopub.status.busy": "2024-06-04T23:19:10.800548Z",
     "iopub.status.idle": "2024-06-04T23:19:10.805188Z",
     "shell.execute_reply": "2024-06-04T23:19:10.804964Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "scaler.fit(feature_df)\n",
    "X_std = scaler.transform(feature_df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5d9c875",
   "metadata": {},
   "source": [
    "Now every column of `feature_std` below has a standard deviation of\n",
    "one and a mean of zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "d1e40190",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.806480Z",
     "iopub.status.busy": "2024-06-04T23:19:10.806406Z",
     "iopub.status.idle": "2024-06-04T23:19:10.810671Z",
     "shell.execute_reply": "2024-06-04T23:19:10.810474Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MOSTYPE     1.000086\n",
       "MAANTHUI    1.000086\n",
       "MGEMOMV     1.000086\n",
       "MGEMLEEF    1.000086\n",
       "MOSHOOFD    1.000086\n",
       "              ...   \n",
       "AZEILPL     1.000086\n",
       "APLEZIER    1.000086\n",
       "AFIETS      1.000086\n",
       "AINBOED     1.000086\n",
       "ABYSTAND    1.000086\n",
       "Length: 85, dtype: float64"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_std = pd.DataFrame(\n",
    "                 X_std,\n",
    "                 columns=feature_df.columns);\n",
    "feature_std.std()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c225f2b2",
   "metadata": {},
   "source": [
    "Notice that the standard deviations are not quite $1$ here; this is again due to some procedures using the $1/n$ convention for variances (in this case `scaler()`), while others use $1/(n-1)$ (the `std()` method). See the footnote on page~\\pageref{Ch4-varformula}.\n",
    "In this case it does not matter, as long as the variables are all on the same scale.\n",
    "\n",
    "Using the function `train_test_split()`  we now split the observations into a test set,\n",
    "containing 1000 observations, and a training set containing the remaining\n",
    "observations. The argument `random_state=0` ensures that we get\n",
    "the same split each time we rerun the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "44ff90d4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.811997Z",
     "iopub.status.busy": "2024-06-04T23:19:10.811926Z",
     "iopub.status.idle": "2024-06-04T23:19:10.814896Z",
     "shell.execute_reply": "2024-06-04T23:19:10.814681Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "(X_train,\n",
    " X_test,\n",
    " y_train,\n",
    " y_test) = train_test_split(np.asarray(feature_std),\n",
    "                            Purchase,\n",
    "                            test_size=1000,\n",
    "                            random_state=0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "293eaa56",
   "metadata": {},
   "source": [
    "`?train_test_split` reveals that the non-keyword arguments can be `lists`, `arrays`, `pandas dataframes` etc that all have the same length (`shape[0]`) and hence are *indexable*. In this case they are the dataframe `feature_std` and the response variable `Purchase`.\n",
    " {Note that we have converted `feature_std` to an `ndarray` to address a bug in `sklearn`.}\n",
    "We fit a KNN model on the training data using $K=1$,\n",
    "and evaluate its performance on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "f88990de",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.816100Z",
     "iopub.status.busy": "2024-06-04T23:19:10.816032Z",
     "iopub.status.idle": "2024-06-04T23:19:10.977519Z",
     "shell.execute_reply": "2024-06-04T23:19:10.973692Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.111, 0.067)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn1 = KNeighborsClassifier(n_neighbors=1)\n",
    "knn1_pred = knn1.fit(X_train, y_train).predict(X_test)\n",
    "np.mean(y_test != knn1_pred), np.mean(y_test != \"No\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57a4a331",
   "metadata": {},
   "source": [
    "The KNN error rate on the 1,000 test observations is about $11%$.\n",
    "At first glance, this may appear to be fairly good. However, since\n",
    "just over 6% of customers purchased insurance, we could get the error\n",
    "rate down to almost 6% by always predicting `No` regardless of the\n",
    "values of the predictors! This is known as the *null rate*.}\n",
    "\n",
    "Suppose that there is some non-trivial cost to trying to sell\n",
    "insurance to a given individual. For instance, perhaps a salesperson\n",
    "must visit each potential customer.  If the company tries to sell\n",
    "insurance to a random selection of customers, then the success rate\n",
    "will be only 6%, which may be far too low given the costs\n",
    "involved.  Instead, the company would like to try to sell insurance\n",
    "only to customers who are likely to buy it. So the overall error rate\n",
    "is not of interest. Instead, the fraction of individuals that are\n",
    "correctly predicted to buy insurance is of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "733b69fb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:10.986636Z",
     "iopub.status.busy": "2024-06-04T23:19:10.986365Z",
     "iopub.status.idle": "2024-06-04T23:19:11.006708Z",
     "shell.execute_reply": "2024-06-04T23:19:11.001541Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Truth</th>\n",
       "      <th>No</th>\n",
       "      <th>Yes</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predicted</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>No</th>\n",
       "      <td>880</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>53</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Truth       No  Yes\n",
       "Predicted          \n",
       "No         880   58\n",
       "Yes         53    9"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_table(knn1_pred, y_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7a1359d",
   "metadata": {},
   "source": [
    "It turns out that KNN with $K=1$ does far better than random guessing\n",
    "among the customers that are predicted to buy insurance. Among 62\n",
    "such customers, 9, or 14.5%, actually do purchase insurance.\n",
    "This is double the rate that one would obtain from random guessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "269d3d95",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.019124Z",
     "iopub.status.busy": "2024-06-04T23:19:11.016874Z",
     "iopub.status.idle": "2024-06-04T23:19:11.026784Z",
     "shell.execute_reply": "2024-06-04T23:19:11.025439Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.14516129032258066"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "9/(53+9)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12bcb480",
   "metadata": {},
   "source": [
    "### Tuning Parameters\n",
    "\n",
    "The number of neighbors in KNN is referred to as a *tuning parameter*, also referred to as a *hyperparameter*.\n",
    "We do not know *a priori* what value to use. It is therefore of interest\n",
    "to see how the classifier performs on test data as we vary these\n",
    "parameters. This can be achieved with a `for` loop, described in Section~\\ref{Ch2-statlearn-lab:for-loops}.\n",
    "Here we use a for loop to look at the accuracy of our classifier in the group predicted to purchase\n",
    "insurance as we vary the number of neighbors from 1 to 5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "db9963d8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.029587Z",
     "iopub.status.busy": "2024-06-04T23:19:11.029376Z",
     "iopub.status.idle": "2024-06-04T23:19:11.164487Z",
     "shell.execute_reply": "2024-06-04T23:19:11.164119Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "K=1: # predicted to rent: 62,  # who did rent 9, accuracy 14.5%\n",
      "K=2: # predicted to rent:  6,  # who did rent 1, accuracy 16.7%\n",
      "K=3: # predicted to rent: 20,  # who did rent 3, accuracy 15.0%\n",
      "K=4: # predicted to rent:  4,  # who did rent 0, accuracy 0.0%\n",
      "K=5: # predicted to rent:  7,  # who did rent 1, accuracy 14.3%\n"
     ]
    }
   ],
   "source": [
    "for K in range(1,6):\n",
    "    knn = KNeighborsClassifier(n_neighbors=K)\n",
    "    knn_pred = knn.fit(X_train, y_train).predict(X_test)\n",
    "    C = confusion_table(knn_pred, y_test)\n",
    "    templ = ('K={0:d}: # predicted to rent: {1:>2},' +\n",
    "            '  # who did rent {2:d}, accuracy {3:.1%}')\n",
    "    pred = C.loc['Yes'].sum()\n",
    "    did_rent = C.loc['Yes','Yes']\n",
    "    print(templ.format(\n",
    "          K,\n",
    "          pred,\n",
    "          did_rent,\n",
    "          did_rent / pred))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d3a4b95",
   "metadata": {},
   "source": [
    "We see some variability ---  the numbers for `K=4` are very different from the rest.\n",
    "\n",
    "### Comparison to Logistic Regression\n",
    "As a comparison, we can also fit a logistic regression model to the\n",
    "data. This can also be done\n",
    "with `sklearn`, though by default it fits\n",
    "something like the *ridge regression* version\n",
    "of logistic regression, which we introduce in Chapter~\\ref{Ch6:varselect}. This can\n",
    "be modified by appropriately setting the argument `C` below. Its default\n",
    "value is 1 but by setting it to a very large number, the algorithm converges to the same solution as the usual (unregularized)\n",
    "logistic regression estimator discussed above.\n",
    "\n",
    "Unlike the\n",
    "`statsmodels` package, `sklearn` focuses less on\n",
    "inference and more on classification. Hence,\n",
    "the `summary` methods seen in `statsmodels`\n",
    "and our simplified version seen with `summarize` are not\n",
    "generally available for the classifiers in `sklearn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "77f8eb90",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.166394Z",
     "iopub.status.busy": "2024-06-04T23:19:11.166286Z",
     "iopub.status.idle": "2024-06-04T23:19:11.612761Z",
     "shell.execute_reply": "2024-06-04T23:19:11.611650Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Truth</th>\n",
       "      <th>No</th>\n",
       "      <th>Yes</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predicted</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>No</th>\n",
       "      <td>931</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Truth       No  Yes\n",
       "Predicted          \n",
       "No         931   67\n",
       "Yes          2    0"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logit = LogisticRegression(C=1e10, solver='liblinear')\n",
    "logit.fit(X_train, y_train)\n",
    "logit_pred = logit.predict_proba(X_test)\n",
    "logit_labels = np.where(logit_pred[:,1] > .5, 'Yes', 'No')\n",
    "confusion_table(logit_labels, y_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbf84545",
   "metadata": {},
   "source": [
    "We used the argument `solver='liblinear'` above to\n",
    "avoid a warning with the default solver which would indicate that\n",
    "the algorithm does not converge.\n",
    "\n",
    "If we use $0.5$ as the predicted probability cut-off for the\n",
    "classifier, then we have a problem: only two of the test observations\n",
    "are predicted to purchase insurance.  However, we are not required to use a\n",
    "cut-off of $0.5$. If we instead predict a purchase any time the\n",
    "predicted probability of purchase exceeds $0.25$, we get much better\n",
    "results: we predict that 29 people will purchase insurance, and we are\n",
    "correct for about 31% of these people. This is almost five times\n",
    "better than random guessing!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "907e3299",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.615999Z",
     "iopub.status.busy": "2024-06-04T23:19:11.615768Z",
     "iopub.status.idle": "2024-06-04T23:19:11.628399Z",
     "shell.execute_reply": "2024-06-04T23:19:11.627025Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Truth</th>\n",
       "      <th>No</th>\n",
       "      <th>Yes</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predicted</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>No</th>\n",
       "      <td>913</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>20</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Truth       No  Yes\n",
       "Predicted          \n",
       "No         913   58\n",
       "Yes         20    9"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logit_labels = np.where(logit_pred[:,1]>0.25, 'Yes', 'No')\n",
    "confusion_table(logit_labels, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "cb3f2b0e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.631843Z",
     "iopub.status.busy": "2024-06-04T23:19:11.631560Z",
     "iopub.status.idle": "2024-06-04T23:19:11.636898Z",
     "shell.execute_reply": "2024-06-04T23:19:11.636444Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3103448275862069"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "9/(20+9)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d3b5503",
   "metadata": {},
   "source": [
    "## Linear and Poisson Regression on the Bikeshare Data\n",
    "Here we fit linear and  Poisson regression models to the `Bikeshare` data, as described in Section~\\ref{Ch4:sec:pois}.\n",
    "The response `bikers` measures the number of bike rentals per hour\n",
    "in Washington, DC in the period 2010--2012."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "23ce05b5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.639885Z",
     "iopub.status.busy": "2024-06-04T23:19:11.639635Z",
     "iopub.status.idle": "2024-06-04T23:19:11.652008Z",
     "shell.execute_reply": "2024-06-04T23:19:11.651483Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "Bike = load_data('Bikeshare')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdb0a62a",
   "metadata": {},
   "source": [
    "Let's have a peek at the dimensions and names of the variables in this dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "027a24c4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.655682Z",
     "iopub.status.busy": "2024-06-04T23:19:11.655146Z",
     "iopub.status.idle": "2024-06-04T23:19:11.661660Z",
     "shell.execute_reply": "2024-06-04T23:19:11.658906Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((8645, 15),\n",
       " Index(['season', 'mnth', 'day', 'hr', 'holiday', 'weekday', 'workingday',\n",
       "        'weathersit', 'temp', 'atemp', 'hum', 'windspeed', 'casual',\n",
       "        'registered', 'bikers'],\n",
       "       dtype='object'))"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Bike.shape, Bike.columns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce51b618",
   "metadata": {},
   "source": [
    "### Linear Regression\n",
    "\n",
    "We begin by fitting a linear regression model to the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "5896ce19",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.664908Z",
     "iopub.status.busy": "2024-06-04T23:19:11.664631Z",
     "iopub.status.idle": "2024-06-04T23:19:11.731259Z",
     "shell.execute_reply": "2024-06-04T23:19:11.730708Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>-68.6317</td>\n",
       "      <td>5.307</td>\n",
       "      <td>-12.932</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[Feb]</th>\n",
       "      <td>6.8452</td>\n",
       "      <td>4.287</td>\n",
       "      <td>1.597</td>\n",
       "      <td>0.110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[March]</th>\n",
       "      <td>16.5514</td>\n",
       "      <td>4.301</td>\n",
       "      <td>3.848</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[April]</th>\n",
       "      <td>41.4249</td>\n",
       "      <td>4.972</td>\n",
       "      <td>8.331</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[May]</th>\n",
       "      <td>72.5571</td>\n",
       "      <td>5.641</td>\n",
       "      <td>12.862</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[June]</th>\n",
       "      <td>67.8187</td>\n",
       "      <td>6.544</td>\n",
       "      <td>10.364</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[July]</th>\n",
       "      <td>45.3245</td>\n",
       "      <td>7.081</td>\n",
       "      <td>6.401</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[Aug]</th>\n",
       "      <td>53.2430</td>\n",
       "      <td>6.640</td>\n",
       "      <td>8.019</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[Sept]</th>\n",
       "      <td>66.6783</td>\n",
       "      <td>5.925</td>\n",
       "      <td>11.254</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[Oct]</th>\n",
       "      <td>75.8343</td>\n",
       "      <td>4.950</td>\n",
       "      <td>15.319</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[Nov]</th>\n",
       "      <td>60.3100</td>\n",
       "      <td>4.610</td>\n",
       "      <td>13.083</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[Dec]</th>\n",
       "      <td>46.4577</td>\n",
       "      <td>4.271</td>\n",
       "      <td>10.878</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[1]</th>\n",
       "      <td>-14.5793</td>\n",
       "      <td>5.699</td>\n",
       "      <td>-2.558</td>\n",
       "      <td>0.011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[2]</th>\n",
       "      <td>-21.5791</td>\n",
       "      <td>5.733</td>\n",
       "      <td>-3.764</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[3]</th>\n",
       "      <td>-31.1408</td>\n",
       "      <td>5.778</td>\n",
       "      <td>-5.389</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[4]</th>\n",
       "      <td>-36.9075</td>\n",
       "      <td>5.802</td>\n",
       "      <td>-6.361</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[5]</th>\n",
       "      <td>-24.1355</td>\n",
       "      <td>5.737</td>\n",
       "      <td>-4.207</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[6]</th>\n",
       "      <td>20.5997</td>\n",
       "      <td>5.704</td>\n",
       "      <td>3.612</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[7]</th>\n",
       "      <td>120.0931</td>\n",
       "      <td>5.693</td>\n",
       "      <td>21.095</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[8]</th>\n",
       "      <td>223.6619</td>\n",
       "      <td>5.690</td>\n",
       "      <td>39.310</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[9]</th>\n",
       "      <td>120.5819</td>\n",
       "      <td>5.693</td>\n",
       "      <td>21.182</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[10]</th>\n",
       "      <td>83.8013</td>\n",
       "      <td>5.705</td>\n",
       "      <td>14.689</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[11]</th>\n",
       "      <td>105.4234</td>\n",
       "      <td>5.722</td>\n",
       "      <td>18.424</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[12]</th>\n",
       "      <td>137.2837</td>\n",
       "      <td>5.740</td>\n",
       "      <td>23.916</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[13]</th>\n",
       "      <td>136.0359</td>\n",
       "      <td>5.760</td>\n",
       "      <td>23.617</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[14]</th>\n",
       "      <td>126.6361</td>\n",
       "      <td>5.776</td>\n",
       "      <td>21.923</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[15]</th>\n",
       "      <td>132.0865</td>\n",
       "      <td>5.780</td>\n",
       "      <td>22.852</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[16]</th>\n",
       "      <td>178.5206</td>\n",
       "      <td>5.772</td>\n",
       "      <td>30.927</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[17]</th>\n",
       "      <td>296.2670</td>\n",
       "      <td>5.749</td>\n",
       "      <td>51.537</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[18]</th>\n",
       "      <td>269.4409</td>\n",
       "      <td>5.736</td>\n",
       "      <td>46.976</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[19]</th>\n",
       "      <td>186.2558</td>\n",
       "      <td>5.714</td>\n",
       "      <td>32.596</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[20]</th>\n",
       "      <td>125.5492</td>\n",
       "      <td>5.704</td>\n",
       "      <td>22.012</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[21]</th>\n",
       "      <td>87.5537</td>\n",
       "      <td>5.693</td>\n",
       "      <td>15.378</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[22]</th>\n",
       "      <td>59.1226</td>\n",
       "      <td>5.689</td>\n",
       "      <td>10.392</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[23]</th>\n",
       "      <td>26.8376</td>\n",
       "      <td>5.688</td>\n",
       "      <td>4.719</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>workingday</th>\n",
       "      <td>1.2696</td>\n",
       "      <td>1.784</td>\n",
       "      <td>0.711</td>\n",
       "      <td>0.477</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>temp</th>\n",
       "      <td>157.2094</td>\n",
       "      <td>10.261</td>\n",
       "      <td>15.321</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weathersit[cloudy/misty]</th>\n",
       "      <td>-12.8903</td>\n",
       "      <td>1.964</td>\n",
       "      <td>-6.562</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weathersit[heavy rain/snow]</th>\n",
       "      <td>-109.7446</td>\n",
       "      <td>76.667</td>\n",
       "      <td>-1.431</td>\n",
       "      <td>0.152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weathersit[light rain/snow]</th>\n",
       "      <td>-66.4944</td>\n",
       "      <td>2.965</td>\n",
       "      <td>-22.425</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 coef  std err       t  P>|t|\n",
       "intercept                    -68.6317    5.307 -12.932  0.000\n",
       "mnth[Feb]                      6.8452    4.287   1.597  0.110\n",
       "mnth[March]                   16.5514    4.301   3.848  0.000\n",
       "mnth[April]                   41.4249    4.972   8.331  0.000\n",
       "mnth[May]                     72.5571    5.641  12.862  0.000\n",
       "mnth[June]                    67.8187    6.544  10.364  0.000\n",
       "mnth[July]                    45.3245    7.081   6.401  0.000\n",
       "mnth[Aug]                     53.2430    6.640   8.019  0.000\n",
       "mnth[Sept]                    66.6783    5.925  11.254  0.000\n",
       "mnth[Oct]                     75.8343    4.950  15.319  0.000\n",
       "mnth[Nov]                     60.3100    4.610  13.083  0.000\n",
       "mnth[Dec]                     46.4577    4.271  10.878  0.000\n",
       "hr[1]                        -14.5793    5.699  -2.558  0.011\n",
       "hr[2]                        -21.5791    5.733  -3.764  0.000\n",
       "hr[3]                        -31.1408    5.778  -5.389  0.000\n",
       "hr[4]                        -36.9075    5.802  -6.361  0.000\n",
       "hr[5]                        -24.1355    5.737  -4.207  0.000\n",
       "hr[6]                         20.5997    5.704   3.612  0.000\n",
       "hr[7]                        120.0931    5.693  21.095  0.000\n",
       "hr[8]                        223.6619    5.690  39.310  0.000\n",
       "hr[9]                        120.5819    5.693  21.182  0.000\n",
       "hr[10]                        83.8013    5.705  14.689  0.000\n",
       "hr[11]                       105.4234    5.722  18.424  0.000\n",
       "hr[12]                       137.2837    5.740  23.916  0.000\n",
       "hr[13]                       136.0359    5.760  23.617  0.000\n",
       "hr[14]                       126.6361    5.776  21.923  0.000\n",
       "hr[15]                       132.0865    5.780  22.852  0.000\n",
       "hr[16]                       178.5206    5.772  30.927  0.000\n",
       "hr[17]                       296.2670    5.749  51.537  0.000\n",
       "hr[18]                       269.4409    5.736  46.976  0.000\n",
       "hr[19]                       186.2558    5.714  32.596  0.000\n",
       "hr[20]                       125.5492    5.704  22.012  0.000\n",
       "hr[21]                        87.5537    5.693  15.378  0.000\n",
       "hr[22]                        59.1226    5.689  10.392  0.000\n",
       "hr[23]                        26.8376    5.688   4.719  0.000\n",
       "workingday                     1.2696    1.784   0.711  0.477\n",
       "temp                         157.2094   10.261  15.321  0.000\n",
       "weathersit[cloudy/misty]     -12.8903    1.964  -6.562  0.000\n",
       "weathersit[heavy rain/snow] -109.7446   76.667  -1.431  0.152\n",
       "weathersit[light rain/snow]  -66.4944    2.965 -22.425  0.000"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = MS(['mnth',\n",
    "        'hr',\n",
    "        'workingday',\n",
    "        'temp',\n",
    "        'weathersit']).fit_transform(Bike)\n",
    "Y = Bike['bikers']\n",
    "M_lm = sm.OLS(Y, X).fit()\n",
    "summarize(M_lm)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d165f13c",
   "metadata": {},
   "source": [
    "There are 24 levels in `hr` and 40 rows in all.\n",
    "In `M_lm`, the first levels `hr[0]` and `mnth[Jan]`  are treated\n",
    "as the baseline values, and so no coefficient estimates are provided\n",
    "for them: implicitly, their coefficient estimates are zero, and all\n",
    "other levels are measured relative to these baselines. For example,\n",
    "the Feb coefficient of $6.845$ signifies that, holding all other\n",
    "variables constant, there are on average about 7 more riders in\n",
    "February than in January. Similarly there are about 16.5 more riders\n",
    "in March than in January.\n",
    "\n",
    "The results seen in Section~\\ref{sec:bikeshare.linear}\n",
    "used a slightly different coding of the variables `hr` and `mnth`, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "3b8a24b4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.734388Z",
     "iopub.status.busy": "2024-06-04T23:19:11.734170Z",
     "iopub.status.idle": "2024-06-04T23:19:11.738116Z",
     "shell.execute_reply": "2024-06-04T23:19:11.737083Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "hr_encode = contrast('hr', 'sum')\n",
    "mnth_encode = contrast('mnth', 'sum')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee108ad7",
   "metadata": {},
   "source": [
    "Refitting again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "276ec935",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.741132Z",
     "iopub.status.busy": "2024-06-04T23:19:11.740955Z",
     "iopub.status.idle": "2024-06-04T23:19:11.806258Z",
     "shell.execute_reply": "2024-06-04T23:19:11.805740Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>73.5974</td>\n",
       "      <td>5.132</td>\n",
       "      <td>14.340</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[Jan]</th>\n",
       "      <td>-46.0871</td>\n",
       "      <td>4.085</td>\n",
       "      <td>-11.281</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[Feb]</th>\n",
       "      <td>-39.2419</td>\n",
       "      <td>3.539</td>\n",
       "      <td>-11.088</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[March]</th>\n",
       "      <td>-29.5357</td>\n",
       "      <td>3.155</td>\n",
       "      <td>-9.361</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[April]</th>\n",
       "      <td>-4.6622</td>\n",
       "      <td>2.741</td>\n",
       "      <td>-1.701</td>\n",
       "      <td>0.089</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[May]</th>\n",
       "      <td>26.4700</td>\n",
       "      <td>2.851</td>\n",
       "      <td>9.285</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[June]</th>\n",
       "      <td>21.7317</td>\n",
       "      <td>3.465</td>\n",
       "      <td>6.272</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[July]</th>\n",
       "      <td>-0.7626</td>\n",
       "      <td>3.908</td>\n",
       "      <td>-0.195</td>\n",
       "      <td>0.845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[Aug]</th>\n",
       "      <td>7.1560</td>\n",
       "      <td>3.535</td>\n",
       "      <td>2.024</td>\n",
       "      <td>0.043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[Sept]</th>\n",
       "      <td>20.5912</td>\n",
       "      <td>3.046</td>\n",
       "      <td>6.761</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[Oct]</th>\n",
       "      <td>29.7472</td>\n",
       "      <td>2.700</td>\n",
       "      <td>11.019</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mnth[Nov]</th>\n",
       "      <td>14.2229</td>\n",
       "      <td>2.860</td>\n",
       "      <td>4.972</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[0]</th>\n",
       "      <td>-96.1420</td>\n",
       "      <td>3.955</td>\n",
       "      <td>-24.307</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[1]</th>\n",
       "      <td>-110.7213</td>\n",
       "      <td>3.966</td>\n",
       "      <td>-27.916</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[2]</th>\n",
       "      <td>-117.7212</td>\n",
       "      <td>4.016</td>\n",
       "      <td>-29.310</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[3]</th>\n",
       "      <td>-127.2828</td>\n",
       "      <td>4.081</td>\n",
       "      <td>-31.191</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[4]</th>\n",
       "      <td>-133.0495</td>\n",
       "      <td>4.117</td>\n",
       "      <td>-32.319</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[5]</th>\n",
       "      <td>-120.2775</td>\n",
       "      <td>4.037</td>\n",
       "      <td>-29.794</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[6]</th>\n",
       "      <td>-75.5424</td>\n",
       "      <td>3.992</td>\n",
       "      <td>-18.925</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[7]</th>\n",
       "      <td>23.9511</td>\n",
       "      <td>3.969</td>\n",
       "      <td>6.035</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[8]</th>\n",
       "      <td>127.5199</td>\n",
       "      <td>3.950</td>\n",
       "      <td>32.284</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[9]</th>\n",
       "      <td>24.4399</td>\n",
       "      <td>3.936</td>\n",
       "      <td>6.209</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[10]</th>\n",
       "      <td>-12.3407</td>\n",
       "      <td>3.936</td>\n",
       "      <td>-3.135</td>\n",
       "      <td>0.002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[11]</th>\n",
       "      <td>9.2814</td>\n",
       "      <td>3.945</td>\n",
       "      <td>2.353</td>\n",
       "      <td>0.019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[12]</th>\n",
       "      <td>41.1417</td>\n",
       "      <td>3.957</td>\n",
       "      <td>10.397</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[13]</th>\n",
       "      <td>39.8939</td>\n",
       "      <td>3.975</td>\n",
       "      <td>10.036</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[14]</th>\n",
       "      <td>30.4940</td>\n",
       "      <td>3.991</td>\n",
       "      <td>7.641</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[15]</th>\n",
       "      <td>35.9445</td>\n",
       "      <td>3.995</td>\n",
       "      <td>8.998</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[16]</th>\n",
       "      <td>82.3786</td>\n",
       "      <td>3.988</td>\n",
       "      <td>20.655</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[17]</th>\n",
       "      <td>200.1249</td>\n",
       "      <td>3.964</td>\n",
       "      <td>50.488</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[18]</th>\n",
       "      <td>173.2989</td>\n",
       "      <td>3.956</td>\n",
       "      <td>43.806</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[19]</th>\n",
       "      <td>90.1138</td>\n",
       "      <td>3.940</td>\n",
       "      <td>22.872</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[20]</th>\n",
       "      <td>29.4071</td>\n",
       "      <td>3.936</td>\n",
       "      <td>7.471</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[21]</th>\n",
       "      <td>-8.5883</td>\n",
       "      <td>3.933</td>\n",
       "      <td>-2.184</td>\n",
       "      <td>0.029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hr[22]</th>\n",
       "      <td>-37.0194</td>\n",
       "      <td>3.934</td>\n",
       "      <td>-9.409</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>workingday</th>\n",
       "      <td>1.2696</td>\n",
       "      <td>1.784</td>\n",
       "      <td>0.711</td>\n",
       "      <td>0.477</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>temp</th>\n",
       "      <td>157.2094</td>\n",
       "      <td>10.261</td>\n",
       "      <td>15.321</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weathersit[cloudy/misty]</th>\n",
       "      <td>-12.8903</td>\n",
       "      <td>1.964</td>\n",
       "      <td>-6.562</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weathersit[heavy rain/snow]</th>\n",
       "      <td>-109.7446</td>\n",
       "      <td>76.667</td>\n",
       "      <td>-1.431</td>\n",
       "      <td>0.152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weathersit[light rain/snow]</th>\n",
       "      <td>-66.4944</td>\n",
       "      <td>2.965</td>\n",
       "      <td>-22.425</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 coef  std err       t  P>|t|\n",
       "intercept                     73.5974    5.132  14.340  0.000\n",
       "mnth[Jan]                    -46.0871    4.085 -11.281  0.000\n",
       "mnth[Feb]                    -39.2419    3.539 -11.088  0.000\n",
       "mnth[March]                  -29.5357    3.155  -9.361  0.000\n",
       "mnth[April]                   -4.6622    2.741  -1.701  0.089\n",
       "mnth[May]                     26.4700    2.851   9.285  0.000\n",
       "mnth[June]                    21.7317    3.465   6.272  0.000\n",
       "mnth[July]                    -0.7626    3.908  -0.195  0.845\n",
       "mnth[Aug]                      7.1560    3.535   2.024  0.043\n",
       "mnth[Sept]                    20.5912    3.046   6.761  0.000\n",
       "mnth[Oct]                     29.7472    2.700  11.019  0.000\n",
       "mnth[Nov]                     14.2229    2.860   4.972  0.000\n",
       "hr[0]                        -96.1420    3.955 -24.307  0.000\n",
       "hr[1]                       -110.7213    3.966 -27.916  0.000\n",
       "hr[2]                       -117.7212    4.016 -29.310  0.000\n",
       "hr[3]                       -127.2828    4.081 -31.191  0.000\n",
       "hr[4]                       -133.0495    4.117 -32.319  0.000\n",
       "hr[5]                       -120.2775    4.037 -29.794  0.000\n",
       "hr[6]                        -75.5424    3.992 -18.925  0.000\n",
       "hr[7]                         23.9511    3.969   6.035  0.000\n",
       "hr[8]                        127.5199    3.950  32.284  0.000\n",
       "hr[9]                         24.4399    3.936   6.209  0.000\n",
       "hr[10]                       -12.3407    3.936  -3.135  0.002\n",
       "hr[11]                         9.2814    3.945   2.353  0.019\n",
       "hr[12]                        41.1417    3.957  10.397  0.000\n",
       "hr[13]                        39.8939    3.975  10.036  0.000\n",
       "hr[14]                        30.4940    3.991   7.641  0.000\n",
       "hr[15]                        35.9445    3.995   8.998  0.000\n",
       "hr[16]                        82.3786    3.988  20.655  0.000\n",
       "hr[17]                       200.1249    3.964  50.488  0.000\n",
       "hr[18]                       173.2989    3.956  43.806  0.000\n",
       "hr[19]                        90.1138    3.940  22.872  0.000\n",
       "hr[20]                        29.4071    3.936   7.471  0.000\n",
       "hr[21]                        -8.5883    3.933  -2.184  0.029\n",
       "hr[22]                       -37.0194    3.934  -9.409  0.000\n",
       "workingday                     1.2696    1.784   0.711  0.477\n",
       "temp                         157.2094   10.261  15.321  0.000\n",
       "weathersit[cloudy/misty]     -12.8903    1.964  -6.562  0.000\n",
       "weathersit[heavy rain/snow] -109.7446   76.667  -1.431  0.152\n",
       "weathersit[light rain/snow]  -66.4944    2.965 -22.425  0.000"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X2 = MS([mnth_encode,\n",
    "         hr_encode,\n",
    "        'workingday',\n",
    "        'temp',\n",
    "        'weathersit']).fit_transform(Bike)\n",
    "M2_lm = sm.OLS(Y, X2).fit()\n",
    "S2 = summarize(M2_lm)\n",
    "S2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7e31352",
   "metadata": {},
   "source": [
    "What is the difference between the two codings?  In `M2_lm`, a\n",
    "coefficient estimate is reported for all but level `23` of `hr`\n",
    "and level `Dec` of `mnth`. Importantly, in `M2_lm`, the (unreported) coefficient estimate\n",
    "for the last level of `mnth` is not zero: instead, it equals the\n",
    "negative of the sum of the coefficient estimates for all of the\n",
    "other levels. Similarly, in `M2_lm`, the coefficient estimate\n",
    "for the last level of `hr` is the negative of the sum of the\n",
    "coefficient estimates for all of the other levels. This means that the\n",
    "coefficients of `hr` and `mnth` in `M2_lm` will always sum\n",
    "to zero, and can be interpreted as the difference from the mean\n",
    "level. For example, the coefficient for January of $-46.087$ indicates\n",
    "that, holding all other variables constant, there are typically 46\n",
    "fewer riders in January relative to the yearly average.\n",
    "\n",
    "It is important to realize that the choice of coding really does not\n",
    "matter, provided that we interpret the  model output correctly in light\n",
    "of the coding used. For example, we see that the predictions from the\n",
    "linear model are the same regardless of coding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "c8b43ab6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.812237Z",
     "iopub.status.busy": "2024-06-04T23:19:11.811979Z",
     "iopub.status.idle": "2024-06-04T23:19:11.819745Z",
     "shell.execute_reply": "2024-06-04T23:19:11.819144Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0334731385542263e-18"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum((M_lm.fittedvalues - M2_lm.fittedvalues)**2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f17733ac",
   "metadata": {},
   "source": [
    "The sum of squared differences is zero. We can also see this using the\n",
    "`np.allclose()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "bcc538c2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.823477Z",
     "iopub.status.busy": "2024-06-04T23:19:11.823254Z",
     "iopub.status.idle": "2024-06-04T23:19:11.830397Z",
     "shell.execute_reply": "2024-06-04T23:19:11.829852Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(M_lm.fittedvalues, M2_lm.fittedvalues)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41fb2787",
   "metadata": {},
   "source": [
    "To reproduce the left-hand side of Figure~\\ref{Ch4:bikeshare}\n",
    "we must first obtain the coefficient estimates associated with\n",
    "`mnth`. The coefficients for January through November can be obtained\n",
    "directly from the `M2_lm` object. The coefficient for December\n",
    "must be explicitly computed as the negative sum of all the other\n",
    "months. We first extract all the coefficients for month from\n",
    "the coefficients of `M2_lm`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "33a75971",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.833582Z",
     "iopub.status.busy": "2024-06-04T23:19:11.833376Z",
     "iopub.status.idle": "2024-06-04T23:19:11.840363Z",
     "shell.execute_reply": "2024-06-04T23:19:11.839611Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "mnth[Jan]     -46.0871\n",
       "mnth[Feb]     -39.2419\n",
       "mnth[March]   -29.5357\n",
       "mnth[April]    -4.6622\n",
       "mnth[May]      26.4700\n",
       "mnth[June]     21.7317\n",
       "mnth[July]     -0.7626\n",
       "mnth[Aug]       7.1560\n",
       "mnth[Sept]     20.5912\n",
       "mnth[Oct]      29.7472\n",
       "mnth[Nov]      14.2229\n",
       "Name: coef, dtype: float64"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coef_month = S2[S2.index.str.contains('mnth')]['coef']\n",
    "coef_month\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7caad472",
   "metadata": {},
   "source": [
    "Next, we append `Dec` as the negative of the sum of all other months."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "eeac39db",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.843459Z",
     "iopub.status.busy": "2024-06-04T23:19:11.843230Z",
     "iopub.status.idle": "2024-06-04T23:19:11.850225Z",
     "shell.execute_reply": "2024-06-04T23:19:11.849688Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "mnth[Jan]     -46.0871\n",
       "mnth[Feb]     -39.2419\n",
       "mnth[March]   -29.5357\n",
       "mnth[April]    -4.6622\n",
       "mnth[May]      26.4700\n",
       "mnth[June]     21.7317\n",
       "mnth[July]     -0.7626\n",
       "mnth[Aug]       7.1560\n",
       "mnth[Sept]     20.5912\n",
       "mnth[Oct]      29.7472\n",
       "mnth[Nov]      14.2229\n",
       "mnth[Dec]       0.3705\n",
       "dtype: float64"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "months = Bike['mnth'].dtype.categories\n",
    "coef_month = pd.concat([\n",
    "                       coef_month,\n",
    "                       pd.Series([-coef_month.sum()],\n",
    "                                  index=['mnth[Dec]'\n",
    "                                 ])\n",
    "                       ])\n",
    "coef_month\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55a5ff99",
   "metadata": {},
   "source": [
    "Finally, to make the plot neater, we’ll just use the first letter of each month, which is the $6$th entry of each of\n",
    "the labels in the index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "e86a3652",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.853511Z",
     "iopub.status.busy": "2024-06-04T23:19:11.853047Z",
     "iopub.status.idle": "2024-06-04T23:19:11.977554Z",
     "shell.execute_reply": "2024-06-04T23:19:11.976958Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 49608 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig_month, ax_month = subplots(figsize=(8,8))\n",
    "x_month = np.arange(coef_month.shape[0])\n",
    "ax_month.plot(x_month, coef_month, marker='o', ms=10)\n",
    "ax_month.set_xticks(x_month)\n",
    "ax_month.set_xticklabels([l[5] for l in coef_month.index], fontsize=20)\n",
    "ax_month.set_xlabel('Month', fontsize=20)\n",
    "ax_month.set_ylabel('Coefficient', fontsize=20);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c68761a",
   "metadata": {},
   "source": [
    "Reproducing the  right-hand plot in Figure~\\ref{Ch4:bikeshare}  follows a similar process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "5abb32ed",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.980369Z",
     "iopub.status.busy": "2024-06-04T23:19:11.980198Z",
     "iopub.status.idle": "2024-06-04T23:19:11.985705Z",
     "shell.execute_reply": "2024-06-04T23:19:11.984785Z"
    }
   },
   "outputs": [],
   "source": [
    "coef_hr = S2[S2.index.str.contains('hr')]['coef']\n",
    "coef_hr = coef_hr.reindex(['hr[{0}]'.format(h) for h in range(23)])\n",
    "coef_hr = pd.concat([coef_hr,\n",
    "                     pd.Series([-coef_hr.sum()], index=['hr[23]'])\n",
    "                    ])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc51083b",
   "metadata": {},
   "source": [
    "We now make the hour plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "0f5698be",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:11.989339Z",
     "iopub.status.busy": "2024-06-04T23:19:11.988415Z",
     "iopub.status.idle": "2024-06-04T23:19:12.111821Z",
     "shell.execute_reply": "2024-06-04T23:19:12.111533Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 56228 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig_hr, ax_hr = subplots(figsize=(8,8))\n",
    "x_hr = np.arange(coef_hr.shape[0])\n",
    "ax_hr.plot(x_hr, coef_hr, marker='o', ms=10)\n",
    "ax_hr.set_xticks(x_hr[::2])\n",
    "ax_hr.set_xticklabels(range(24)[::2], fontsize=20)\n",
    "ax_hr.set_xlabel('Hour', fontsize=20)\n",
    "ax_hr.set_ylabel('Coefficient', fontsize=20);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c43958c3",
   "metadata": {},
   "source": [
    "### Poisson Regression\n",
    "\n",
    "Now we fit instead a Poisson regression model to the\n",
    "`Bikeshare` data. Very little changes, except that we now use the\n",
    "function `sm.GLM()` with the Poisson family specified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "1262ac4f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:12.113270Z",
     "iopub.status.busy": "2024-06-04T23:19:12.113151Z",
     "iopub.status.idle": "2024-06-04T23:19:12.187994Z",
     "shell.execute_reply": "2024-06-04T23:19:12.187097Z"
    }
   },
   "outputs": [],
   "source": [
    "M_pois = sm.GLM(Y, X2, family=sm.families.Poisson()).fit()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8552fb8b",
   "metadata": {},
   "source": [
    "We can plot the coefficients associated with `mnth` and `hr`, in order to reproduce  Figure~\\ref{Ch4:bikeshare.pois}. We first complete these coefficients as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "feeea491",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:12.191417Z",
     "iopub.status.busy": "2024-06-04T23:19:12.191189Z",
     "iopub.status.idle": "2024-06-04T23:19:12.211795Z",
     "shell.execute_reply": "2024-06-04T23:19:12.211208Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "S_pois = summarize(M_pois)\n",
    "coef_month = S_pois[S_pois.index.str.contains('mnth')]['coef']\n",
    "coef_month = pd.concat([coef_month,\n",
    "                        pd.Series([-coef_month.sum()],\n",
    "                                   index=['mnth[Dec]'])])\n",
    "coef_hr = S_pois[S_pois.index.str.contains('hr')]['coef']\n",
    "coef_hr = pd.concat([coef_hr,\n",
    "                     pd.Series([-coef_hr.sum()],\n",
    "                     index=['hr[23]'])])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a52b9a03",
   "metadata": {},
   "source": [
    "The plotting is as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "09926adc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:12.215056Z",
     "iopub.status.busy": "2024-06-04T23:19:12.214817Z",
     "iopub.status.idle": "2024-06-04T23:19:12.405617Z",
     "shell.execute_reply": "2024-06-04T23:19:12.405120Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 82252 characters]",
      "text/plain": [
       "<Figure size 1600x800 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig_pois, (ax_month, ax_hr) = subplots(1, 2, figsize=(16,8))\n",
    "ax_month.plot(x_month, coef_month, marker='o', ms=10)\n",
    "ax_month.set_xticks(x_month)\n",
    "ax_month.set_xticklabels([l[5] for l in coef_month.index], fontsize=20)\n",
    "ax_month.set_xlabel('Month', fontsize=20)\n",
    "ax_month.set_ylabel('Coefficient', fontsize=20)\n",
    "ax_hr.plot(x_hr, coef_hr, marker='o', ms=10)\n",
    "ax_hr.set_xticklabels(range(24)[::2], fontsize=20)\n",
    "ax_hr.set_xlabel('Hour', fontsize=20)\n",
    "ax_hr.set_ylabel('Coefficient', fontsize=20);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "812fed8c",
   "metadata": {},
   "source": [
    "We compare the fitted values of the two models.\n",
    "The fitted values are stored in the `fittedvalues` attribute\n",
    "returned by the `fit()` method for both the linear regression and the Poisson\n",
    "fits. The linear predictors are stored as the attribute `lin_pred`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "bf9e9f1d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-04T23:19:12.409134Z",
     "iopub.status.busy": "2024-06-04T23:19:12.408900Z",
     "iopub.status.idle": "2024-06-04T23:19:12.505338Z",
     "shell.execute_reply": "2024-06-04T23:19:12.505080Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 77916 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = subplots(figsize=(8, 8))\n",
    "ax.scatter(M2_lm.fittedvalues,\n",
    "           M_pois.fittedvalues,\n",
    "           s=20)\n",
    "ax.set_xlabel('Linear Regression Fit', fontsize=20)\n",
    "ax.set_ylabel('Poisson Regression Fit', fontsize=20)\n",
    "ax.axline([0,0], c='black', linewidth=3,\n",
    "          linestyle='--', slope=1);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc683c95",
   "metadata": {},
   "source": [
    "The predictions from the Poisson regression model are correlated with\n",
    "those from the linear model; however, the former are non-negative. As\n",
    "a result the Poisson regression predictions tend to be larger than\n",
    "those from the linear model for either very low or very high levels of\n",
    "ridership.\n",
    "\n",
    "In this section, we fit Poisson regression models using the `sm.GLM()` function with the argument\n",
    "`family=sm.families.Poisson()`. Earlier in this lab we used the `sm.GLM()` function\n",
    "with `family=sm.families.Binomial()` to perform logistic regression. Other\n",
    "choices for the `family` argument can be used to fit other types\n",
    "of GLMs. For instance, `family=sm.families.Gamma()` fits a Gamma regression\n",
    "model.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
