{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "18156290",
   "metadata": {},
   "source": [
    "\n",
    "# Linear Regression\n",
    "\n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/intro-stat-learning/ISLP_labs/blob/v2.2/Ch03-linreg-lab.ipynb\">\n",
    "<img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "\n",
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/intro-stat-learning/ISLP_labs/v2.2?labpath=Ch03-linreg-lab.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c62aa70",
   "metadata": {},
   "source": [
    "## Importing packages\n",
    "We import our standard libraries at this top\n",
    "level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c5fcfd8f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:52.664224Z",
     "iopub.status.busy": "2025-04-03T04:43:52.663745Z",
     "iopub.status.idle": "2025-04-03T04:43:55.080525Z",
     "shell.execute_reply": "2025-04-03T04:43:55.080187Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from matplotlib.pyplot import subplots\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc3e353a",
   "metadata": {},
   "source": [
    "### New imports\n",
    "Throughout this lab we will introduce new functions and libraries. However,\n",
    "we will import them here to emphasize these are the new\n",
    "code objects in this lab. Keeping imports near the top\n",
    "of a notebook makes the code more readable, since scanning the first few\n",
    "lines tells us what libraries are used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "42cfb62e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:55.082578Z",
     "iopub.status.busy": "2025-04-03T04:43:55.082413Z",
     "iopub.status.idle": "2025-04-03T04:43:58.413315Z",
     "shell.execute_reply": "2025-04-03T04:43:58.412991Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "import statsmodels.api as sm\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd7ce405",
   "metadata": {},
   "source": [
    " We will provide relevant details about the\n",
    "functions below as they are needed.\n",
    "\n",
    "Besides importing whole modules, it is also possible\n",
    "to import only a few items from a given module. This\n",
    "will help keep the  *namespace* clean.\n",
    "We will use a few specific objects from the `statsmodels` package\n",
    "which we import here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b387d3ac",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:58.415310Z",
     "iopub.status.busy": "2025-04-03T04:43:58.415163Z",
     "iopub.status.idle": "2025-04-03T04:43:58.417748Z",
     "shell.execute_reply": "2025-04-03T04:43:58.417483Z"
    }
   },
   "outputs": [],
   "source": [
    "from statsmodels.stats.outliers_influence \\\n",
    "     import variance_inflation_factor as VIF\n",
    "from statsmodels.stats.anova import anova_lm\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4713b12",
   "metadata": {},
   "source": [
    "As one of the import statements above is quite a long line, we inserted a line break `\\` to\n",
    "ease readability.\n",
    "\n",
    "We will also use some functions written for the labs in this book in the `ISLP`\n",
    "package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "307b7e22",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:58.419369Z",
     "iopub.status.busy": "2025-04-03T04:43:58.419266Z",
     "iopub.status.idle": "2025-04-03T04:43:59.495961Z",
     "shell.execute_reply": "2025-04-03T04:43:59.495556Z"
    }
   },
   "outputs": [],
   "source": [
    "from ISLP import load_data\n",
    "from ISLP.models import (ModelSpec as MS,\n",
    "                         summarize,\n",
    "                         poly)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81d778fb",
   "metadata": {},
   "source": [
    "### Inspecting Objects and Namespaces\n",
    "The\n",
    "function  `dir()`\n",
    "provides a list of\n",
    "objects in a namespace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "79099069",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.498234Z",
     "iopub.status.busy": "2025-04-03T04:43:59.497985Z",
     "iopub.status.idle": "2025-04-03T04:43:59.501527Z",
     "shell.execute_reply": "2025-04-03T04:43:59.501281Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['In',\n",
       " 'MS',\n",
       " 'Out',\n",
       " 'VIF',\n",
       " '_',\n",
       " '__',\n",
       " '___',\n",
       " '__builtin__',\n",
       " '__builtins__',\n",
       " '__doc__',\n",
       " '__loader__',\n",
       " '__name__',\n",
       " '__package__',\n",
       " '__spec__',\n",
       " '_dh',\n",
       " '_i',\n",
       " '_i1',\n",
       " '_i2',\n",
       " '_i3',\n",
       " '_i4',\n",
       " '_i5',\n",
       " '_ih',\n",
       " '_ii',\n",
       " '_iii',\n",
       " '_oh',\n",
       " 'anova_lm',\n",
       " 'exit',\n",
       " 'get_ipython',\n",
       " 'load_data',\n",
       " 'np',\n",
       " 'open',\n",
       " 'pd',\n",
       " 'poly',\n",
       " 'quit',\n",
       " 'sm',\n",
       " 'subplots',\n",
       " 'summarize']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "237f9711",
   "metadata": {},
   "source": [
    " This shows you everything that `Python` can find at the top level.\n",
    "There are certain objects like `__builtins__` that contain references to built-in\n",
    "functions like `print()`.\n",
    "\n",
    "Every python object has its own notion of\n",
    "namespace, also accessible with `dir()`. This will include\n",
    "both the attributes of the object\n",
    "as well as any methods associated with it. For instance, we see `'sum'` in the listing for an\n",
    "array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3f99195f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.503342Z",
     "iopub.status.busy": "2025-04-03T04:43:59.503233Z",
     "iopub.status.idle": "2025-04-03T04:43:59.507028Z",
     "shell.execute_reply": "2025-04-03T04:43:59.506764Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['T',\n",
       " '__abs__',\n",
       " '__add__',\n",
       " '__and__',\n",
       " '__array__',\n",
       " '__array_finalize__',\n",
       " '__array_function__',\n",
       " '__array_interface__',\n",
       " '__array_prepare__',\n",
       " '__array_priority__',\n",
       " '__array_struct__',\n",
       " '__array_ufunc__',\n",
       " '__array_wrap__',\n",
       " '__bool__',\n",
       " '__class__',\n",
       " '__class_getitem__',\n",
       " '__complex__',\n",
       " '__contains__',\n",
       " '__copy__',\n",
       " '__deepcopy__',\n",
       " '__delattr__',\n",
       " '__delitem__',\n",
       " '__dir__',\n",
       " '__divmod__',\n",
       " '__dlpack__',\n",
       " '__dlpack_device__',\n",
       " '__doc__',\n",
       " '__eq__',\n",
       " '__float__',\n",
       " '__floordiv__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__getattribute__',\n",
       " '__getitem__',\n",
       " '__getstate__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__iadd__',\n",
       " '__iand__',\n",
       " '__ifloordiv__',\n",
       " '__ilshift__',\n",
       " '__imatmul__',\n",
       " '__imod__',\n",
       " '__imul__',\n",
       " '__index__',\n",
       " '__init__',\n",
       " '__init_subclass__',\n",
       " '__int__',\n",
       " '__invert__',\n",
       " '__ior__',\n",
       " '__ipow__',\n",
       " '__irshift__',\n",
       " '__isub__',\n",
       " '__iter__',\n",
       " '__itruediv__',\n",
       " '__ixor__',\n",
       " '__le__',\n",
       " '__len__',\n",
       " '__lshift__',\n",
       " '__lt__',\n",
       " '__matmul__',\n",
       " '__mod__',\n",
       " '__mul__',\n",
       " '__ne__',\n",
       " '__neg__',\n",
       " '__new__',\n",
       " '__or__',\n",
       " '__pos__',\n",
       " '__pow__',\n",
       " '__radd__',\n",
       " '__rand__',\n",
       " '__rdivmod__',\n",
       " '__reduce__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__rfloordiv__',\n",
       " '__rlshift__',\n",
       " '__rmatmul__',\n",
       " '__rmod__',\n",
       " '__rmul__',\n",
       " '__ror__',\n",
       " '__rpow__',\n",
       " '__rrshift__',\n",
       " '__rshift__',\n",
       " '__rsub__',\n",
       " '__rtruediv__',\n",
       " '__rxor__',\n",
       " '__setattr__',\n",
       " '__setitem__',\n",
       " '__setstate__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__sub__',\n",
       " '__subclasshook__',\n",
       " '__truediv__',\n",
       " '__xor__',\n",
       " 'all',\n",
       " 'any',\n",
       " 'argmax',\n",
       " 'argmin',\n",
       " 'argpartition',\n",
       " 'argsort',\n",
       " 'astype',\n",
       " 'base',\n",
       " 'byteswap',\n",
       " 'choose',\n",
       " 'clip',\n",
       " 'compress',\n",
       " 'conj',\n",
       " 'conjugate',\n",
       " 'copy',\n",
       " 'ctypes',\n",
       " 'cumprod',\n",
       " 'cumsum',\n",
       " 'data',\n",
       " 'diagonal',\n",
       " 'dot',\n",
       " 'dtype',\n",
       " 'dump',\n",
       " 'dumps',\n",
       " 'fill',\n",
       " 'flags',\n",
       " 'flat',\n",
       " 'flatten',\n",
       " 'getfield',\n",
       " 'imag',\n",
       " 'item',\n",
       " 'itemset',\n",
       " 'itemsize',\n",
       " 'max',\n",
       " 'mean',\n",
       " 'min',\n",
       " 'nbytes',\n",
       " 'ndim',\n",
       " 'newbyteorder',\n",
       " 'nonzero',\n",
       " 'partition',\n",
       " 'prod',\n",
       " 'ptp',\n",
       " 'put',\n",
       " 'ravel',\n",
       " 'real',\n",
       " 'repeat',\n",
       " 'reshape',\n",
       " 'resize',\n",
       " 'round',\n",
       " 'searchsorted',\n",
       " 'setfield',\n",
       " 'setflags',\n",
       " 'shape',\n",
       " 'size',\n",
       " 'sort',\n",
       " 'squeeze',\n",
       " 'std',\n",
       " 'strides',\n",
       " 'sum',\n",
       " 'swapaxes',\n",
       " 'take',\n",
       " 'tobytes',\n",
       " 'tofile',\n",
       " 'tolist',\n",
       " 'tostring',\n",
       " 'trace',\n",
       " 'transpose',\n",
       " 'var',\n",
       " 'view']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = np.array([3,5,11])\n",
    "dir(A)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d776f61",
   "metadata": {},
   "source": [
    " This indicates that the object `A.sum` exists. In this case it is a method\n",
    "that can be used to compute the sum of the array `A` as can be seen by typing `A.sum?`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f3f913ca",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.508536Z",
     "iopub.status.busy": "2025-04-03T04:43:59.508431Z",
     "iopub.status.idle": "2025-04-03T04:43:59.510630Z",
     "shell.execute_reply": "2025-04-03T04:43:59.510369Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.sum()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "974fad08",
   "metadata": {},
   "source": [
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b212f1f",
   "metadata": {},
   "source": [
    "## Simple Linear Regression\n",
    "In this section we will  construct model \n",
    "matrices (also called design matrices) using the `ModelSpec()`  transform from `ISLP.models`.\n",
    "\n",
    "We  will use the `Boston` housing data set, which is contained in the `ISLP` package.  The `Boston` dataset records  `medv`  (median house value) for $506$ neighborhoods\n",
    "around Boston.  We will build a regression model to predict  `medv`  using $13$\n",
    "predictors such as  `rmvar`  (average number of rooms per house),\n",
    " `age`  (proportion of owner-occupied units built prior to 1940), and  `lstat`  (percent of\n",
    "households with low socioeconomic status).  We will use `statsmodels` for this\n",
    "task, a `Python` package that implements several commonly used\n",
    "regression methods.\n",
    "\n",
    "We have included a simple loading function `load_data()` in the\n",
    "`ISLP` package:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d720e2c5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.512240Z",
     "iopub.status.busy": "2025-04-03T04:43:59.512127Z",
     "iopub.status.idle": "2025-04-03T04:43:59.518900Z",
     "shell.execute_reply": "2025-04-03T04:43:59.518650Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['crim', 'zn', 'indus', 'chas', 'nox', 'rm', 'age', 'dis', 'rad', 'tax',\n",
       "       'ptratio', 'lstat', 'medv'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Boston = load_data(\"Boston\")\n",
    "Boston.columns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27b9c8f5",
   "metadata": {},
   "source": [
    "Type `Boston?` to find out more about these data.\n",
    "\n",
    "We start by using the `sm.OLS()`  function to fit a\n",
    "simple linear regression model.  Our response will be\n",
    " `medv`  and  `lstat`  will be the single predictor.\n",
    "For this model, we can create the model matrix by hand.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "40e9b0d5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.520460Z",
     "iopub.status.busy": "2025-04-03T04:43:59.520346Z",
     "iopub.status.idle": "2025-04-03T04:43:59.525618Z",
     "shell.execute_reply": "2025-04-03T04:43:59.525347Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>intercept</th>\n",
       "      <th>lstat</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>9.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.94</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   intercept  lstat\n",
       "0        1.0   4.98\n",
       "1        1.0   9.14\n",
       "2        1.0   4.03\n",
       "3        1.0   2.94"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = pd.DataFrame({'intercept': np.ones(Boston.shape[0]),\n",
    "                  'lstat': Boston['lstat']})\n",
    "X[:4]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "162d93ef",
   "metadata": {},
   "source": [
    "We extract the response, and fit the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c1d544d6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.527107Z",
     "iopub.status.busy": "2025-04-03T04:43:59.527008Z",
     "iopub.status.idle": "2025-04-03T04:43:59.529257Z",
     "shell.execute_reply": "2025-04-03T04:43:59.529030Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "y = Boston['medv']\n",
    "model = sm.OLS(y, X)\n",
    "results = model.fit()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a1d62f2",
   "metadata": {},
   "source": [
    "Note that `sm.OLS()` does\n",
    "not fit the model; it specifies the model, and then `model.fit()` does the actual fitting.  \n",
    "\n",
    "Our `ISLP` function `summarize()` produces a simple table of the parameter estimates,\n",
    "their standard errors, t-statistics and p-values.\n",
    "The function takes a single argument, such as the object `results` \n",
    "returned here by the `fit`\n",
    "method, and returns such a summary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3d340a07",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.530690Z",
     "iopub.status.busy": "2025-04-03T04:43:59.530586Z",
     "iopub.status.idle": "2025-04-03T04:43:59.834480Z",
     "shell.execute_reply": "2025-04-03T04:43:59.834171Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>34.5538</td>\n",
       "      <td>0.563</td>\n",
       "      <td>61.415</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lstat</th>\n",
       "      <td>-0.9500</td>\n",
       "      <td>0.039</td>\n",
       "      <td>-24.528</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              coef  std err       t  P>|t|\n",
       "intercept  34.5538    0.563  61.415    0.0\n",
       "lstat      -0.9500    0.039 -24.528    0.0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summarize(results)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31d6b202",
   "metadata": {},
   "source": [
    "Before we describe other methods for working with fitted models, we outline a more useful and general framework for constructing a model matrix~`X`.\n",
    "### Using Transformations: Fit and Transform\n",
    "Our model above has a single predictor, and constructing `X` was straightforward. \n",
    "In practice  we often fit models with more than one predictor, typically selected from an array or data frame.\n",
    "We may wish to introduce transformations to the variables before fitting the model, specify interactions between variables, and expand some particular variables into sets of variables (e.g. polynomials).\n",
    "The `sklearn`  package has a particular notion\n",
    "for this type of task: a *transform*. A transform is an object\n",
    "that is created with some parameters as arguments. The\n",
    "object has two main methods: `fit()` and `transform()`.\n",
    "\n",
    "We provide a general approach for specifying models and constructing\n",
    "the model matrix through the transform `ModelSpec()` in the `ISLP` library.\n",
    "`ModelSpec()`\n",
    "(renamed `MS()` in the preamble) creates a\n",
    "transform object, and then a pair of methods\n",
    "`transform()` and `fit()` are used to construct a\n",
    "corresponding model matrix.\n",
    "\n",
    "We first describe this process for our simple regression model  using a single predictor `lstat` in\n",
    "the `Boston` data frame, but will use it repeatedly in more\n",
    "complex tasks in this and other labs in this book.\n",
    "In our case the transform is created by the expression\n",
    "`design = MS(['lstat'])`.\n",
    "\n",
    "The `fit()`  method takes the original array and may do some\n",
    "initial computations on it, as specified in the transform object.\n",
    "For example, it may compute means and standard deviations for centering and scaling.\n",
    "The `transform()` \n",
    "method applies the fitted transformation to the array of data, and produces the model matrix.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "11b07af0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.836170Z",
     "iopub.status.busy": "2025-04-03T04:43:59.836022Z",
     "iopub.status.idle": "2025-04-03T04:43:59.842946Z",
     "shell.execute_reply": "2025-04-03T04:43:59.842669Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>intercept</th>\n",
       "      <th>lstat</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>9.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.94</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   intercept  lstat\n",
       "0        1.0   4.98\n",
       "1        1.0   9.14\n",
       "2        1.0   4.03\n",
       "3        1.0   2.94"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "design = MS(['lstat'])\n",
    "design = design.fit(Boston)\n",
    "X = design.transform(Boston)\n",
    "X[:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f35e5292",
   "metadata": {},
   "source": [
    "In this simple case, the `fit()`  method does very little; it simply checks that the variable `'lstat'` specified in `design` exists in `Boston`. Then `transform()` constructs the model matrix with two columns: an `intercept` and the variable `lstat`.\n",
    "\n",
    "These two operations can be  combined with the\n",
    "`fit_transform()`  method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d5c7b370",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.844509Z",
     "iopub.status.busy": "2025-04-03T04:43:59.844405Z",
     "iopub.status.idle": "2025-04-03T04:43:59.850962Z",
     "shell.execute_reply": "2025-04-03T04:43:59.850679Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>intercept</th>\n",
       "      <th>lstat</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>9.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.94</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   intercept  lstat\n",
       "0        1.0   4.98\n",
       "1        1.0   9.14\n",
       "2        1.0   4.03\n",
       "3        1.0   2.94"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "design = MS(['lstat'])\n",
    "X = design.fit_transform(Boston)\n",
    "X[:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f09689d",
   "metadata": {},
   "source": [
    "Note that, as in the previous code chunk when the two steps were done separately, the `design` object is changed as a result of the `fit()` operation. The power of this pipeline will become clearer when we fit more complex models that involve interactions and transformations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94a7c0aa",
   "metadata": {},
   "source": [
    "Let's return to our fitted regression model.\n",
    "The object\n",
    "`results` has several methods that can be used for inference.\n",
    "We already presented a function `summarize()` for showing the essentials of the fit.\n",
    "For a full and somewhat exhaustive summary of the fit, we can use the `summary()` \n",
    "method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5b205c0b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.852617Z",
     "iopub.status.busy": "2025-04-03T04:43:59.852508Z",
     "iopub.status.idle": "2025-04-03T04:43:59.862779Z",
     "shell.execute_reply": "2025-04-03T04:43:59.862528Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>          <td>medv</td>       <th>  R-squared:         </th> <td>   0.544</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.543</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   601.6</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Wed, 02 Apr 2025</td> <th>  Prob (F-statistic):</th> <td>5.08e-88</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>21:43:59</td>     <th>  Log-Likelihood:    </th> <td> -1641.5</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>   506</td>      <th>  AIC:               </th> <td>   3287.</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>   504</td>      <th>  BIC:               </th> <td>   3295.</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     1</td>      <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   34.5538</td> <td>    0.563</td> <td>   61.415</td> <td> 0.000</td> <td>   33.448</td> <td>   35.659</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>lstat</th>     <td>   -0.9500</td> <td>    0.039</td> <td>  -24.528</td> <td> 0.000</td> <td>   -1.026</td> <td>   -0.874</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>137.043</td> <th>  Durbin-Watson:     </th> <td>   0.892</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th> <td> 0.000</td>  <th>  Jarque-Bera (JB):  </th> <td> 291.373</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>          <td> 1.453</td>  <th>  Prob(JB):          </th> <td>5.36e-64</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>      <td> 5.319</td>  <th>  Cond. No.          </th> <td>    29.7</td>\n",
       "</tr>\n",
       "</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
      ],
      "text/latex": [
       "\\begin{center}\n",
       "\\begin{tabular}{lclc}\n",
       "\\toprule\n",
       "\\textbf{Dep. Variable:}    &       medv       & \\textbf{  R-squared:         } &     0.544   \\\\\n",
       "\\textbf{Model:}            &       OLS        & \\textbf{  Adj. R-squared:    } &     0.543   \\\\\n",
       "\\textbf{Method:}           &  Least Squares   & \\textbf{  F-statistic:       } &     601.6   \\\\\n",
       "\\textbf{Date:}             & Wed, 02 Apr 2025 & \\textbf{  Prob (F-statistic):} &  5.08e-88   \\\\\n",
       "\\textbf{Time:}             &     21:43:59     & \\textbf{  Log-Likelihood:    } &   -1641.5   \\\\\n",
       "\\textbf{No. Observations:} &         506      & \\textbf{  AIC:               } &     3287.   \\\\\n",
       "\\textbf{Df Residuals:}     &         504      & \\textbf{  BIC:               } &     3295.   \\\\\n",
       "\\textbf{Df Model:}         &           1      & \\textbf{                     } &             \\\\\n",
       "\\textbf{Covariance Type:}  &    nonrobust     & \\textbf{                     } &             \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lcccccc}\n",
       "                   & \\textbf{coef} & \\textbf{std err} & \\textbf{t} & \\textbf{P$> |$t$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\\n",
       "\\midrule\n",
       "\\textbf{intercept} &      34.5538  &        0.563     &    61.415  &         0.000        &       33.448    &       35.659     \\\\\n",
       "\\textbf{lstat}     &      -0.9500  &        0.039     &   -24.528  &         0.000        &       -1.026    &       -0.874     \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "\\begin{tabular}{lclc}\n",
       "\\textbf{Omnibus:}       & 137.043 & \\textbf{  Durbin-Watson:     } &    0.892  \\\\\n",
       "\\textbf{Prob(Omnibus):} &   0.000 & \\textbf{  Jarque-Bera (JB):  } &  291.373  \\\\\n",
       "\\textbf{Skew:}          &   1.453 & \\textbf{  Prob(JB):          } & 5.36e-64  \\\\\n",
       "\\textbf{Kurtosis:}      &   5.319 & \\textbf{  Cond. No.          } &     29.7  \\\\\n",
       "\\bottomrule\n",
       "\\end{tabular}\n",
       "%\\caption{OLS Regression Results}\n",
       "\\end{center}\n",
       "\n",
       "Notes: \\newline\n",
       " [1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:                   medv   R-squared:                       0.544\n",
       "Model:                            OLS   Adj. R-squared:                  0.543\n",
       "Method:                 Least Squares   F-statistic:                     601.6\n",
       "Date:                Wed, 02 Apr 2025   Prob (F-statistic):           5.08e-88\n",
       "Time:                        21:43:59   Log-Likelihood:                -1641.5\n",
       "No. Observations:                 506   AIC:                             3287.\n",
       "Df Residuals:                     504   BIC:                             3295.\n",
       "Df Model:                           1                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "==============================================================================\n",
       "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     34.5538      0.563     61.415      0.000      33.448      35.659\n",
       "lstat         -0.9500      0.039    -24.528      0.000      -1.026      -0.874\n",
       "==============================================================================\n",
       "Omnibus:                      137.043   Durbin-Watson:                   0.892\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):              291.373\n",
       "Skew:                           1.453   Prob(JB):                     5.36e-64\n",
       "Kurtosis:                       5.319   Cond. No.                         29.7\n",
       "==============================================================================\n",
       "\n",
       "Notes:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "\"\"\""
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.summary()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c30baa4",
   "metadata": {},
   "source": [
    "The fitted coefficients can also be retrieved as the\n",
    "`params` attribute of `results`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4b024467",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.864311Z",
     "iopub.status.busy": "2025-04-03T04:43:59.864203Z",
     "iopub.status.idle": "2025-04-03T04:43:59.866715Z",
     "shell.execute_reply": "2025-04-03T04:43:59.866467Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "intercept    34.553841\n",
       "lstat        -0.950049\n",
       "dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.params\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb7bbd4c",
   "metadata": {},
   "source": [
    "The `get_prediction()`  method can be used to obtain predictions, and produce confidence intervals and\n",
    "prediction intervals for the prediction of  `medv`  for  given values of  `lstat`.\n",
    "\n",
    "We first create a new data frame, in this case containing only the variable `lstat`, with the values for this variable at which we wish to make predictions.\n",
    "We then use the `transform()` method of `design` to create the corresponding model matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "195e0ace",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.868150Z",
     "iopub.status.busy": "2025-04-03T04:43:59.868043Z",
     "iopub.status.idle": "2025-04-03T04:43:59.871887Z",
     "shell.execute_reply": "2025-04-03T04:43:59.871636Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>intercept</th>\n",
       "      <th>lstat</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   intercept  lstat\n",
       "0        1.0      5\n",
       "1        1.0     10\n",
       "2        1.0     15"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df = pd.DataFrame({'lstat':[5, 10, 15]})\n",
    "newX = design.transform(new_df)\n",
    "newX\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43565f15",
   "metadata": {},
   "source": [
    "Next we compute the predictions at `newX`, and view them by extracting the `predicted_mean` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "768374f6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.873365Z",
     "iopub.status.busy": "2025-04-03T04:43:59.873277Z",
     "iopub.status.idle": "2025-04-03T04:43:59.875776Z",
     "shell.execute_reply": "2025-04-03T04:43:59.875518Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([29.80359411, 25.05334734, 20.30310057])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_predictions = results.get_prediction(newX);\n",
    "new_predictions.predicted_mean\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "313c1720",
   "metadata": {},
   "source": [
    "We can produce confidence intervals for the predicted values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1a86b943",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.877236Z",
     "iopub.status.busy": "2025-04-03T04:43:59.877150Z",
     "iopub.status.idle": "2025-04-03T04:43:59.879505Z",
     "shell.execute_reply": "2025-04-03T04:43:59.879266Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[29.00741194, 30.59977628],\n",
       "       [24.47413202, 25.63256267],\n",
       "       [19.73158815, 20.87461299]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_predictions.conf_int(alpha=0.05)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3595344e",
   "metadata": {},
   "source": [
    "Prediction intervals are computing by setting `obs=True`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "ea9727ca",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.880882Z",
     "iopub.status.busy": "2025-04-03T04:43:59.880804Z",
     "iopub.status.idle": "2025-04-03T04:43:59.883099Z",
     "shell.execute_reply": "2025-04-03T04:43:59.882842Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[17.56567478, 42.04151344],\n",
       "       [12.82762635, 37.27906833],\n",
       "       [ 8.0777421 , 32.52845905]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_predictions.conf_int(obs=True, alpha=0.05)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b157334d",
   "metadata": {},
   "source": [
    " For instance, the 95% confidence interval associated with an\n",
    " `lstat`  value of 10 is (24.47, 25.63), and the 95% prediction\n",
    "interval is (12.82, 37.28).  As expected, the confidence and\n",
    "prediction intervals are centered around the same point (a predicted\n",
    "value of 25.05 for  `medv`  when  `lstat`  equals\n",
    "10), but the latter are substantially wider.\n",
    "\n",
    "Next we will plot  `medv`  and  `lstat` \n",
    "using `DataFrame.plot.scatter()`, \\definelongblankMR{plot.scatter()}{plot.slashslashscatter()}\n",
    "and wish to\n",
    "add the regression line to the resulting plot."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05f5b29a",
   "metadata": {},
   "source": [
    "### Defining Functions\n",
    "While there is a function\n",
    "within the `ISLP` package that adds a line to an existing plot, we take this opportunity\n",
    "to define our first function to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "796c20f6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.884628Z",
     "iopub.status.busy": "2025-04-03T04:43:59.884549Z",
     "iopub.status.idle": "2025-04-03T04:43:59.886382Z",
     "shell.execute_reply": "2025-04-03T04:43:59.886149Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "def abline(ax, b, m):\n",
    "    \"Add a line with slope m and intercept b to ax\"\n",
    "    xlim = ax.get_xlim()\n",
    "    ylim = [m * xlim[0] + b, m * xlim[1] + b]\n",
    "    ax.plot(xlim, ylim)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5eb99e73",
   "metadata": {},
   "source": [
    " A few things are illustrated above. First we see the syntax for defining a function:\n",
    "`def funcname(...)`. The function has arguments `ax, b, m`\n",
    "where `ax` is an axis object for an exisiting plot, `b` is the intercept and\n",
    "`m` is the slope of the desired line. Other plotting  options can be passed on to\n",
    "`ax.plot` by including additional optional arguments as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a258b8c0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.887825Z",
     "iopub.status.busy": "2025-04-03T04:43:59.887719Z",
     "iopub.status.idle": "2025-04-03T04:43:59.889620Z",
     "shell.execute_reply": "2025-04-03T04:43:59.889379Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "def abline(ax, b, m, *args, **kwargs):\n",
    "    \"Add a line with slope m and intercept b to ax\"\n",
    "    xlim = ax.get_xlim()\n",
    "    ylim = [m * xlim[0] + b, m * xlim[1] + b]\n",
    "    ax.plot(xlim, ylim, *args, **kwargs)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be8f4615",
   "metadata": {},
   "source": [
    "The addition of `*args` allows any number of\n",
    "non-named arguments to `abline`, while `*kwargs` allows any\n",
    "number of named arguments (such as `linewidth=3`) to `abline`.\n",
    "In our function, we pass\n",
    "these arguments verbatim to `ax.plot` above. Readers\n",
    "interested in learning more about\n",
    "functions are referred to the section on\n",
    "defining functions in [docs.python.org/tutorial](https://docs.python.org/3/tutorial/controlflow.html#defining-functions).\n",
    "\n",
    "Let’s use our new function to add this regression line to a plot of\n",
    "`medv` vs. `lstat`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "56b75491",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:43:59.891025Z",
     "iopub.status.busy": "2025-04-03T04:43:59.890922Z",
     "iopub.status.idle": "2025-04-03T04:44:00.051132Z",
     "shell.execute_reply": "2025-04-03T04:44:00.050413Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 39980 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = Boston.plot.scatter('lstat', 'medv')\n",
    "abline(ax,\n",
    "       results.params[0],\n",
    "       results.params[1],\n",
    "       'r--',\n",
    "       linewidth=3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04a1e56f",
   "metadata": {},
   "source": [
    "Thus, the final call to `ax.plot()` is `ax.plot(xlim, ylim, 'r--', linewidth=3)`.\n",
    "We have used the argument `'r--'` to produce a red dashed line, and added\n",
    "an argument to make it of width 3.\n",
    "There is some evidence for non-linearity in the relationship between  `lstat`  and  `medv`. We will explore this issue later in this lab.\n",
    "\n",
    "As mentioned above, there is an existing function to add a line to a plot --- `ax.axline()` --- but knowing how to write such functions empowers us to create more expressive displays.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a083307",
   "metadata": {},
   "source": [
    "Next we examine some diagnostic plots, several of which were discussed\n",
    "in Section~\\ref{Ch3:problems.sec}.\n",
    "We can find the fitted values and residuals\n",
    "of the fit as attributes of the `results` object.\n",
    "Various influence measures describing the regression model\n",
    "are computed with the `get_influence()` method.\n",
    "As we will not use the `fig` component returned\n",
    "as the first value from `subplots()`, we simply\n",
    "capture the second returned value in `ax` below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "74be2100",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.056158Z",
     "iopub.status.busy": "2025-04-03T04:44:00.055868Z",
     "iopub.status.idle": "2025-04-03T04:44:00.204012Z",
     "shell.execute_reply": "2025-04-03T04:44:00.203709Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 54088 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = subplots(figsize=(8,8))[1]\n",
    "ax.scatter(results.fittedvalues, results.resid)\n",
    "ax.set_xlabel('Fitted value')\n",
    "ax.set_ylabel('Residual')\n",
    "ax.axhline(0, c='k', ls='--');\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c00db5a1",
   "metadata": {},
   "source": [
    " We add a horizontal line at 0 for reference using the\n",
    " `ax.axhline()`   method, indicating\n",
    "it should be black (`c='k'`) and have a dashed linestyle (`ls='--'`).\n",
    "\n",
    "On the basis of the residual plot, there is some evidence of non-linearity.\n",
    "Leverage statistics can be computed for any number of predictors using the\n",
    "`hat_matrix_diag` attribute of the value returned by the\n",
    "`get_influence()`  method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "db8bc925",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.205721Z",
     "iopub.status.busy": "2025-04-03T04:44:00.205598Z",
     "iopub.status.idle": "2025-04-03T04:44:00.291490Z",
     "shell.execute_reply": "2025-04-03T04:44:00.291204Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "374"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 47020 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "infl = results.get_influence()\n",
    "ax = subplots(figsize=(8,8))[1]\n",
    "ax.scatter(np.arange(X.shape[0]), infl.hat_matrix_diag)\n",
    "ax.set_xlabel('Index')\n",
    "ax.set_ylabel('Leverage')\n",
    "np.argmax(infl.hat_matrix_diag)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e7aa0bb",
   "metadata": {},
   "source": [
    " The `np.argmax()`  function identifies the index of the largest element of an array, optionally computed over an axis of the array.\n",
    "In this case, we maximized over the entire array\n",
    "to determine which observation has the largest leverage statistic."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e98db373",
   "metadata": {},
   "source": [
    "## Multiple Linear Regression\n",
    "In order to fit a multiple linear regression model using least squares, we again use\n",
    "the `ModelSpec()`  transform to construct the required\n",
    "model matrix and response. The arguments\n",
    "to `ModelSpec()` can be quite general, but in this case\n",
    "a list of column names suffice. We consider a fit here with\n",
    "the two variables `lstat` and `age`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "e479373c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.293264Z",
     "iopub.status.busy": "2025-04-03T04:44:00.293146Z",
     "iopub.status.idle": "2025-04-03T04:44:00.305835Z",
     "shell.execute_reply": "2025-04-03T04:44:00.305546Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>33.2228</td>\n",
       "      <td>0.731</td>\n",
       "      <td>45.458</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lstat</th>\n",
       "      <td>-1.0321</td>\n",
       "      <td>0.048</td>\n",
       "      <td>-21.416</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>0.0345</td>\n",
       "      <td>0.012</td>\n",
       "      <td>2.826</td>\n",
       "      <td>0.005</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              coef  std err       t  P>|t|\n",
       "intercept  33.2228    0.731  45.458  0.000\n",
       "lstat      -1.0321    0.048 -21.416  0.000\n",
       "age         0.0345    0.012   2.826  0.005"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = MS(['lstat', 'age']).fit_transform(Boston)\n",
    "model1 = sm.OLS(y, X)\n",
    "results1 = model1.fit()\n",
    "summarize(results1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c79b00d",
   "metadata": {},
   "source": [
    "Notice how we have compacted the first line into a succinct expression describing the construction of `X`.\n",
    "\n",
    "The  `Boston`   data set contains 12 variables, and so it would be cumbersome\n",
    "to have to type all of these in order to perform a regression using all of the predictors.\n",
    "Instead, we can use the following short-hand:\\definelongblankMR{columns.drop()}{columns.slashslashdrop()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b903170c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.308200Z",
     "iopub.status.busy": "2025-04-03T04:44:00.308088Z",
     "iopub.status.idle": "2025-04-03T04:44:00.310943Z",
     "shell.execute_reply": "2025-04-03T04:44:00.310561Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['crim', 'zn', 'indus', 'chas', 'nox', 'rm', 'age', 'dis', 'rad', 'tax',\n",
       "       'ptratio', 'lstat'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "terms = Boston.columns.drop('medv')\n",
    "terms\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "179b7846",
   "metadata": {},
   "source": [
    "We can now fit the model with all the variables in `terms` using\n",
    "the same model matrix builder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "bbe47670",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.312663Z",
     "iopub.status.busy": "2025-04-03T04:44:00.312541Z",
     "iopub.status.idle": "2025-04-03T04:44:00.332188Z",
     "shell.execute_reply": "2025-04-03T04:44:00.331912Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>41.6173</td>\n",
       "      <td>4.936</td>\n",
       "      <td>8.431</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>crim</th>\n",
       "      <td>-0.1214</td>\n",
       "      <td>0.033</td>\n",
       "      <td>-3.678</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zn</th>\n",
       "      <td>0.0470</td>\n",
       "      <td>0.014</td>\n",
       "      <td>3.384</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>indus</th>\n",
       "      <td>0.0135</td>\n",
       "      <td>0.062</td>\n",
       "      <td>0.217</td>\n",
       "      <td>0.829</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chas</th>\n",
       "      <td>2.8400</td>\n",
       "      <td>0.870</td>\n",
       "      <td>3.264</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nox</th>\n",
       "      <td>-18.7580</td>\n",
       "      <td>3.851</td>\n",
       "      <td>-4.870</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rm</th>\n",
       "      <td>3.6581</td>\n",
       "      <td>0.420</td>\n",
       "      <td>8.705</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>0.0036</td>\n",
       "      <td>0.013</td>\n",
       "      <td>0.271</td>\n",
       "      <td>0.787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dis</th>\n",
       "      <td>-1.4908</td>\n",
       "      <td>0.202</td>\n",
       "      <td>-7.394</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rad</th>\n",
       "      <td>0.2894</td>\n",
       "      <td>0.067</td>\n",
       "      <td>4.325</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tax</th>\n",
       "      <td>-0.0127</td>\n",
       "      <td>0.004</td>\n",
       "      <td>-3.337</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ptratio</th>\n",
       "      <td>-0.9375</td>\n",
       "      <td>0.132</td>\n",
       "      <td>-7.091</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lstat</th>\n",
       "      <td>-0.5520</td>\n",
       "      <td>0.051</td>\n",
       "      <td>-10.897</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              coef  std err       t  P>|t|\n",
       "intercept  41.6173    4.936   8.431  0.000\n",
       "crim       -0.1214    0.033  -3.678  0.000\n",
       "zn          0.0470    0.014   3.384  0.001\n",
       "indus       0.0135    0.062   0.217  0.829\n",
       "chas        2.8400    0.870   3.264  0.001\n",
       "nox       -18.7580    3.851  -4.870  0.000\n",
       "rm          3.6581    0.420   8.705  0.000\n",
       "age         0.0036    0.013   0.271  0.787\n",
       "dis        -1.4908    0.202  -7.394  0.000\n",
       "rad         0.2894    0.067   4.325  0.000\n",
       "tax        -0.0127    0.004  -3.337  0.001\n",
       "ptratio    -0.9375    0.132  -7.091  0.000\n",
       "lstat      -0.5520    0.051 -10.897  0.000"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = MS(terms).fit_transform(Boston)\n",
    "model = sm.OLS(y, X)\n",
    "results = model.fit()\n",
    "summarize(results)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "183c3c40",
   "metadata": {},
   "source": [
    "What if we would like to perform a regression using all of the variables but one?  For\n",
    "example, in the above regression output,   `age`  has a high $p$-value.\n",
    "So we may wish to run a regression excluding this predictor.\n",
    "The following syntax results in a regression using all predictors except  `age`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "a1484d00",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.333834Z",
     "iopub.status.busy": "2025-04-03T04:44:00.333742Z",
     "iopub.status.idle": "2025-04-03T04:44:00.353383Z",
     "shell.execute_reply": "2025-04-03T04:44:00.353115Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>41.5251</td>\n",
       "      <td>4.920</td>\n",
       "      <td>8.441</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>crim</th>\n",
       "      <td>-0.1214</td>\n",
       "      <td>0.033</td>\n",
       "      <td>-3.683</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zn</th>\n",
       "      <td>0.0465</td>\n",
       "      <td>0.014</td>\n",
       "      <td>3.379</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>indus</th>\n",
       "      <td>0.0135</td>\n",
       "      <td>0.062</td>\n",
       "      <td>0.217</td>\n",
       "      <td>0.829</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chas</th>\n",
       "      <td>2.8528</td>\n",
       "      <td>0.868</td>\n",
       "      <td>3.287</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nox</th>\n",
       "      <td>-18.4851</td>\n",
       "      <td>3.714</td>\n",
       "      <td>-4.978</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rm</th>\n",
       "      <td>3.6811</td>\n",
       "      <td>0.411</td>\n",
       "      <td>8.951</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dis</th>\n",
       "      <td>-1.5068</td>\n",
       "      <td>0.193</td>\n",
       "      <td>-7.825</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rad</th>\n",
       "      <td>0.2879</td>\n",
       "      <td>0.067</td>\n",
       "      <td>4.322</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tax</th>\n",
       "      <td>-0.0127</td>\n",
       "      <td>0.004</td>\n",
       "      <td>-3.333</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ptratio</th>\n",
       "      <td>-0.9346</td>\n",
       "      <td>0.132</td>\n",
       "      <td>-7.099</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lstat</th>\n",
       "      <td>-0.5474</td>\n",
       "      <td>0.048</td>\n",
       "      <td>-11.483</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              coef  std err       t  P>|t|\n",
       "intercept  41.5251    4.920   8.441  0.000\n",
       "crim       -0.1214    0.033  -3.683  0.000\n",
       "zn          0.0465    0.014   3.379  0.001\n",
       "indus       0.0135    0.062   0.217  0.829\n",
       "chas        2.8528    0.868   3.287  0.001\n",
       "nox       -18.4851    3.714  -4.978  0.000\n",
       "rm          3.6811    0.411   8.951  0.000\n",
       "dis        -1.5068    0.193  -7.825  0.000\n",
       "rad         0.2879    0.067   4.322  0.000\n",
       "tax        -0.0127    0.004  -3.333  0.001\n",
       "ptratio    -0.9346    0.132  -7.099  0.000\n",
       "lstat      -0.5474    0.048 -11.483  0.000"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "minus_age = Boston.columns.drop(['medv', 'age']) \n",
    "Xma = MS(minus_age).fit_transform(Boston)\n",
    "model1 = sm.OLS(y, Xma)\n",
    "summarize(model1.fit())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74ec4654",
   "metadata": {},
   "source": [
    "## Multivariate Goodness of Fit\n",
    "We can access the individual components of `results` by name\n",
    "(`dir(results)` shows us what is available). Hence\n",
    "`results.rsquared` gives us the $R^2$,\n",
    "and\n",
    "`np.sqrt(results.scale)` gives us the RSE.\n",
    "\n",
    "Variance inflation factors (section~\\ref{Ch3:problems.sec}) are sometimes useful\n",
    "to assess the effect of collinearity in the model matrix of a regression model.\n",
    "We will compute the VIFs in our multiple regression fit, and use the opportunity to introduce the idea of *list comprehension*.\n",
    "\n",
    "### List Comprehension\n",
    "Often we encounter a sequence of objects which we would like to transform\n",
    "for some other task. Below, we compute the VIF for each\n",
    "feature in our `X` matrix and produce a data frame\n",
    "whose index agrees with the columns of `X`.\n",
    "The notion of list comprehension can often make such\n",
    "a task easier.\n",
    "\n",
    "List comprehensions are simple and powerful ways to form\n",
    "lists of `Python` objects. The language also supports\n",
    "dictionary and *generator* comprehension, though these are\n",
    "beyond our scope here. Let's look at an example. We compute the VIF for each of the variables\n",
    "in the model matrix `X`, using the function `variance_inflation_factor()`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "46ac946c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.354983Z",
     "iopub.status.busy": "2025-04-03T04:44:00.354867Z",
     "iopub.status.idle": "2025-04-03T04:44:00.361815Z",
     "shell.execute_reply": "2025-04-03T04:44:00.361539Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>vif</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>crim</th>\n",
       "      <td>1.767486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zn</th>\n",
       "      <td>2.298459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>indus</th>\n",
       "      <td>3.987181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chas</th>\n",
       "      <td>1.071168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nox</th>\n",
       "      <td>4.369093</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rm</th>\n",
       "      <td>1.912532</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>3.088232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dis</th>\n",
       "      <td>3.954037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rad</th>\n",
       "      <td>7.445301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tax</th>\n",
       "      <td>9.002158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ptratio</th>\n",
       "      <td>1.797060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lstat</th>\n",
       "      <td>2.870777</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              vif\n",
       "crim     1.767486\n",
       "zn       2.298459\n",
       "indus    3.987181\n",
       "chas     1.071168\n",
       "nox      4.369093\n",
       "rm       1.912532\n",
       "age      3.088232\n",
       "dis      3.954037\n",
       "rad      7.445301\n",
       "tax      9.002158\n",
       "ptratio  1.797060\n",
       "lstat    2.870777"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vals = [VIF(X, i)\n",
    "        for i in range(1, X.shape[1])]\n",
    "vif = pd.DataFrame({'vif':vals},\n",
    "                   index=X.columns[1:])\n",
    "vif\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45a29903",
   "metadata": {},
   "source": [
    "The function `VIF()` takes two arguments: a dataframe or array,\n",
    "and a variable column index. In the code above we call `VIF()` on the fly for all columns in `X`.  \n",
    "We have excluded column 0 above (the intercept), which is not of interest. In this case the VIFs are not that exciting.\n",
    "\n",
    "The object `vals` above could have been constructed with the following for loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "b626ecb9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.363405Z",
     "iopub.status.busy": "2025-04-03T04:44:00.363311Z",
     "iopub.status.idle": "2025-04-03T04:44:00.368568Z",
     "shell.execute_reply": "2025-04-03T04:44:00.368275Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "vals = []\n",
    "for i in range(1, X.values.shape[1]):\n",
    "    vals.append(VIF(X.values, i))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e504ae2",
   "metadata": {},
   "source": [
    "List comprehension allows us to perform such repetitive operations in a more straightforward way.\n",
    "## Interaction Terms\n",
    "It is easy to include interaction terms in a linear model using `ModelSpec()`.\n",
    "Including a tuple `(\"lstat\",\"age\")` tells the model\n",
    "matrix builder to include an interaction term between\n",
    " `lstat`  and  `age`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "8bf77477",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.370113Z",
     "iopub.status.busy": "2025-04-03T04:44:00.370024Z",
     "iopub.status.idle": "2025-04-03T04:44:00.384235Z",
     "shell.execute_reply": "2025-04-03T04:44:00.383955Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>36.0885</td>\n",
       "      <td>1.470</td>\n",
       "      <td>24.553</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lstat</th>\n",
       "      <td>-1.3921</td>\n",
       "      <td>0.167</td>\n",
       "      <td>-8.313</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>-0.0007</td>\n",
       "      <td>0.020</td>\n",
       "      <td>-0.036</td>\n",
       "      <td>0.971</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lstat:age</th>\n",
       "      <td>0.0042</td>\n",
       "      <td>0.002</td>\n",
       "      <td>2.244</td>\n",
       "      <td>0.025</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              coef  std err       t  P>|t|\n",
       "intercept  36.0885    1.470  24.553  0.000\n",
       "lstat      -1.3921    0.167  -8.313  0.000\n",
       "age        -0.0007    0.020  -0.036  0.971\n",
       "lstat:age   0.0042    0.002   2.244  0.025"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = MS(['lstat',\n",
    "        'age',\n",
    "        ('lstat', 'age')]).fit_transform(Boston)\n",
    "model2 = sm.OLS(y, X)\n",
    "summarize(model2.fit())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2323ab9",
   "metadata": {},
   "source": [
    "## Non-linear Transformations of the Predictors\n",
    "The model matrix builder can include terms beyond\n",
    "just column names and interactions. For instance,\n",
    "the `poly()` function supplied in `ISLP` specifies that\n",
    "columns representing polynomial functions\n",
    "of its first argument are added to the model matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "6289c3cb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.385906Z",
     "iopub.status.busy": "2025-04-03T04:44:00.385783Z",
     "iopub.status.idle": "2025-04-03T04:44:00.399387Z",
     "shell.execute_reply": "2025-04-03T04:44:00.399084Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>17.7151</td>\n",
       "      <td>0.781</td>\n",
       "      <td>22.681</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(lstat, degree=2)[0]</th>\n",
       "      <td>-179.2279</td>\n",
       "      <td>6.733</td>\n",
       "      <td>-26.620</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poly(lstat, degree=2)[1]</th>\n",
       "      <td>72.9908</td>\n",
       "      <td>5.482</td>\n",
       "      <td>13.315</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>0.0703</td>\n",
       "      <td>0.011</td>\n",
       "      <td>6.471</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              coef  std err       t  P>|t|\n",
       "intercept                  17.7151    0.781  22.681    0.0\n",
       "poly(lstat, degree=2)[0] -179.2279    6.733 -26.620    0.0\n",
       "poly(lstat, degree=2)[1]   72.9908    5.482  13.315    0.0\n",
       "age                         0.0703    0.011   6.471    0.0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = MS([poly('lstat', degree=2), 'age']).fit_transform(Boston)\n",
    "model3 = sm.OLS(y, X)\n",
    "results3 = model3.fit()\n",
    "summarize(results3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bd67b14",
   "metadata": {},
   "source": [
    "The effectively zero *p*-value associated with the quadratic term\n",
    "(i.e. the third row above) suggests that it leads to an improved model.\n",
    "\n",
    "By default, `poly()` creates a basis matrix for inclusion in the\n",
    "model matrix whose\n",
    "columns are *orthogonal polynomials*, which are designed for stable\n",
    "least squares computations. {Actually, `poly()` is a  wrapper for the workhorse and standalone  function `Poly()` that does the  work in building the model matrix.}\n",
    "Alternatively, had we included an argument\n",
    "`raw=True` in the above call to `poly()`, the basis matrix would consist simply of\n",
    "`lstat` and `lstat**2`. Since either of these bases\n",
    "represent quadratic polynomials, the fitted values  would not\n",
    "change in this case, just the polynomial coefficients.  Also by default, the columns\n",
    "created by `poly()` do not include an intercept column as\n",
    "that is automatically added by `MS()`.\n",
    "\n",
    "We use the `anova_lm()` function to further quantify the extent to which the quadratic fit is\n",
    "superior to the linear fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "b6542ac6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.401031Z",
     "iopub.status.busy": "2025-04-03T04:44:00.400918Z",
     "iopub.status.idle": "2025-04-03T04:44:00.406569Z",
     "shell.execute_reply": "2025-04-03T04:44:00.406300Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>df_resid</th>\n",
       "      <th>ssr</th>\n",
       "      <th>df_diff</th>\n",
       "      <th>ss_diff</th>\n",
       "      <th>F</th>\n",
       "      <th>Pr(&gt;F)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>503.0</td>\n",
       "      <td>19168.128609</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>502.0</td>\n",
       "      <td>14165.613251</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5002.515357</td>\n",
       "      <td>177.278785</td>\n",
       "      <td>7.468491e-35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   df_resid           ssr  df_diff      ss_diff           F        Pr(>F)\n",
       "0     503.0  19168.128609      0.0          NaN         NaN           NaN\n",
       "1     502.0  14165.613251      1.0  5002.515357  177.278785  7.468491e-35"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anova_lm(results1, results3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3685159a",
   "metadata": {},
   "source": [
    "Here `results1` represents the linear submodel containing\n",
    "predictors `lstat` and `age`,\n",
    "while `results3` corresponds to the larger model above  with a quadratic\n",
    "term in `lstat`.\n",
    "The `anova_lm()` function performs a hypothesis test\n",
    "comparing the two models. The null hypothesis is that the quadratic\n",
    "term in the bigger model is not needed, and the alternative hypothesis is that the\n",
    "bigger model is superior. Here the *F*-statistic is 177.28 and\n",
    "the associated *p*-value is zero.\n",
    "In this case the *F*-statistic is the square of the\n",
    "*t*-statistic for the quadratic term in the linear model summary\n",
    "for `results3` --- a consequence of the fact that these nested\n",
    "models differ by one degree of freedom.\n",
    "This provides very clear evidence that the quadratic polynomial in\n",
    "`lstat` improves the linear model.\n",
    "This is not surprising, since earlier we saw evidence for non-linearity in the relationship between `medv`\n",
    "and  `lstat`.\n",
    "\n",
    "The function `anova_lm()` can take more than two nested models\n",
    "as input, in which case it compares every successive pair of models.\n",
    "That also explains why their are `NaN`s in the first row above, since\n",
    "there is no previous model with which to compare the first.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "8e4c6004",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.408066Z",
     "iopub.status.busy": "2025-04-03T04:44:00.407960Z",
     "iopub.status.idle": "2025-04-03T04:44:00.502573Z",
     "shell.execute_reply": "2025-04-03T04:44:00.502232Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 54192 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = subplots(figsize=(8,8))[1]\n",
    "ax.scatter(results3.fittedvalues, results3.resid)\n",
    "ax.set_xlabel('Fitted value')\n",
    "ax.set_ylabel('Residual')\n",
    "ax.axhline(0, c='k', ls='--');\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c7e0661",
   "metadata": {},
   "source": [
    "We see that when the quadratic term is included in the model,\n",
    "there is little discernible pattern in the residuals.\n",
    "In order to create a cubic or higher-degree polynomial fit, we can simply change the degree argument\n",
    "to `poly()`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46a4967d",
   "metadata": {},
   "source": [
    "## Qualitative Predictors\n",
    "Here we use the  `Carseats`  data, which is included in the\n",
    "`ISLP` package. We will  attempt to predict `Sales`\n",
    "(child car seat sales) in 400 locations based on a number of\n",
    "predictors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "a83f4bd2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.504359Z",
     "iopub.status.busy": "2025-04-03T04:44:00.504231Z",
     "iopub.status.idle": "2025-04-03T04:44:00.509266Z",
     "shell.execute_reply": "2025-04-03T04:44:00.508991Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Sales', 'CompPrice', 'Income', 'Advertising', 'Population', 'Price',\n",
       "       'ShelveLoc', 'Age', 'Education', 'Urban', 'US'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Carseats = load_data('Carseats')\n",
    "Carseats.columns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69025ecd",
   "metadata": {},
   "source": [
    "The `Carseats`  \n",
    " data includes qualitative predictors such as\n",
    " `ShelveLoc`, an indicator of the quality of the shelving\n",
    " location --- that is,\n",
    "the  space within a store in which the car seat is displayed. The predictor\n",
    " `ShelveLoc`  takes on three possible values, `Bad`, `Medium`, and `Good`.\n",
    "Given a qualitative variable such as  `ShelveLoc`, `ModelSpec()` generates dummy\n",
    "variables automatically.\n",
    "These variables are often referred to as a *one-hot encoding* of the categorical\n",
    "feature. Their columns sum to one, so to avoid collinearity with an intercept, the first column is dropped. Below we see\n",
    "the column `ShelveLoc[Bad]` has been dropped, since `Bad` is the first level of `ShelveLoc`.\n",
    "Below we fit a multiple regression model that includes some interaction terms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "24efc6f1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-04-03T04:44:00.510850Z",
     "iopub.status.busy": "2025-04-03T04:44:00.510746Z",
     "iopub.status.idle": "2025-04-03T04:44:00.538766Z",
     "shell.execute_reply": "2025-04-03T04:44:00.538446Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>std err</th>\n",
       "      <th>t</th>\n",
       "      <th>P&gt;|t|</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>intercept</th>\n",
       "      <td>6.5756</td>\n",
       "      <td>1.009</td>\n",
       "      <td>6.519</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CompPrice</th>\n",
       "      <td>0.0929</td>\n",
       "      <td>0.004</td>\n",
       "      <td>22.567</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Income</th>\n",
       "      <td>0.0109</td>\n",
       "      <td>0.003</td>\n",
       "      <td>4.183</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Advertising</th>\n",
       "      <td>0.0702</td>\n",
       "      <td>0.023</td>\n",
       "      <td>3.107</td>\n",
       "      <td>0.002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Population</th>\n",
       "      <td>0.0002</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.433</td>\n",
       "      <td>0.665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Price</th>\n",
       "      <td>-0.1008</td>\n",
       "      <td>0.007</td>\n",
       "      <td>-13.549</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ShelveLoc[Good]</th>\n",
       "      <td>4.8487</td>\n",
       "      <td>0.153</td>\n",
       "      <td>31.724</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ShelveLoc[Medium]</th>\n",
       "      <td>1.9533</td>\n",
       "      <td>0.126</td>\n",
       "      <td>15.531</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Age</th>\n",
       "      <td>-0.0579</td>\n",
       "      <td>0.016</td>\n",
       "      <td>-3.633</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Education</th>\n",
       "      <td>-0.0209</td>\n",
       "      <td>0.020</td>\n",
       "      <td>-1.063</td>\n",
       "      <td>0.288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Urban[Yes]</th>\n",
       "      <td>0.1402</td>\n",
       "      <td>0.112</td>\n",
       "      <td>1.247</td>\n",
       "      <td>0.213</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>US[Yes]</th>\n",
       "      <td>-0.1576</td>\n",
       "      <td>0.149</td>\n",
       "      <td>-1.058</td>\n",
       "      <td>0.291</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Income:Advertising</th>\n",
       "      <td>0.0008</td>\n",
       "      <td>0.000</td>\n",
       "      <td>2.698</td>\n",
       "      <td>0.007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Price:Age</th>\n",
       "      <td>0.0001</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.801</td>\n",
       "      <td>0.424</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      coef  std err       t  P>|t|\n",
       "intercept           6.5756    1.009   6.519  0.000\n",
       "CompPrice           0.0929    0.004  22.567  0.000\n",
       "Income              0.0109    0.003   4.183  0.000\n",
       "Advertising         0.0702    0.023   3.107  0.002\n",
       "Population          0.0002    0.000   0.433  0.665\n",
       "Price              -0.1008    0.007 -13.549  0.000\n",
       "ShelveLoc[Good]     4.8487    0.153  31.724  0.000\n",
       "ShelveLoc[Medium]   1.9533    0.126  15.531  0.000\n",
       "Age                -0.0579    0.016  -3.633  0.000\n",
       "Education          -0.0209    0.020  -1.063  0.288\n",
       "Urban[Yes]          0.1402    0.112   1.247  0.213\n",
       "US[Yes]            -0.1576    0.149  -1.058  0.291\n",
       "Income:Advertising  0.0008    0.000   2.698  0.007\n",
       "Price:Age           0.0001    0.000   0.801  0.424"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "allvars = list(Carseats.columns.drop('Sales'))\n",
    "y = Carseats['Sales']\n",
    "final = allvars + [('Income', 'Advertising'),\n",
    "                   ('Price', 'Age')]\n",
    "X = MS(final).fit_transform(Carseats)\n",
    "model = sm.OLS(y, X)\n",
    "summarize(model.fit())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8a76419",
   "metadata": {},
   "source": [
    "In the first line above, we made `allvars` a list, so that we\n",
    "could add the interaction terms two lines down. \n",
    "Our model-matrix builder has created a `ShelveLoc[Good]`\n",
    "dummy variable that takes on a value of 1 if the\n",
    "shelving location is good, and 0 otherwise. It has also created a `ShelveLoc[Medium]`\n",
    "dummy variable that equals 1 if the shelving location is medium, and 0 otherwise.\n",
    "A bad shelving location corresponds to a zero for each of the two dummy variables.\n",
    "The fact that the coefficient for `ShelveLoc[Good]` in the regression output is\n",
    "positive indicates that a good shelving location is associated with high sales (relative to a bad location).\n",
    "And `ShelveLoc[Medium]` has a smaller positive coefficient,\n",
    "indicating that a medium shelving location leads to higher sales than a bad\n",
    "shelving location, but lower sales than a good shelving location.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
