{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0c50ec24",
   "metadata": {},
   "source": [
    "\n",
    "# Chapter 8\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98b86be9",
   "metadata": {},
   "source": [
    "# Lab: Tree-Based Methods\n",
    "We import some of our usual libraries at this top\n",
    "level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5061d7d5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:43.521031Z",
     "iopub.status.busy": "2023-08-21T02:29:43.520759Z",
     "iopub.status.idle": "2023-08-21T02:29:44.874525Z",
     "shell.execute_reply": "2023-08-21T02:29:44.874222Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from matplotlib.pyplot import subplots\n",
    "import sklearn.model_selection as skm\n",
    "from ISLP import load_data, confusion_table\n",
    "from ISLP.models import ModelSpec as MS\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98ffeb1c",
   "metadata": {},
   "source": [
    "We also  collect the new imports\n",
    "needed for this lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "747b056a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:44.876483Z",
     "iopub.status.busy": "2023-08-21T02:29:44.876317Z",
     "iopub.status.idle": "2023-08-21T02:29:44.941466Z",
     "shell.execute_reply": "2023-08-21T02:29:44.941148Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import (DecisionTreeClassifier as DTC,\n",
    "                          DecisionTreeRegressor as DTR,\n",
    "                          plot_tree,\n",
    "                          export_text)\n",
    "from sklearn.metrics import (accuracy_score,\n",
    "                             log_loss)\n",
    "from sklearn.ensemble import \\\n",
    "     (RandomForestRegressor as RF,\n",
    "      GradientBoostingRegressor as GBR)\n",
    "from ISLP.bart import BART\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4feef94d",
   "metadata": {},
   "source": [
    "## Fitting Classification Trees"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b977dbf4",
   "metadata": {},
   "source": [
    "We first use classification trees to analyze the  `Carseats`  data set.\n",
    "In these data, `Sales` is a continuous variable, and so we begin\n",
    "by recoding it as a binary variable. We use the `where()` \n",
    "function to create a variable, called `High`, which takes on a\n",
    "value of `Yes` if the `Sales` variable exceeds 8, and takes\n",
    "on a value of `No` otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a29167fd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:44.943319Z",
     "iopub.status.busy": "2023-08-21T02:29:44.943207Z",
     "iopub.status.idle": "2023-08-21T02:29:44.948609Z",
     "shell.execute_reply": "2023-08-21T02:29:44.948363Z"
    }
   },
   "outputs": [],
   "source": [
    "Carseats = load_data('Carseats')\n",
    "High = np.where(Carseats.Sales > 8,\n",
    "                \"Yes\",\n",
    "                \"No\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9887c34b",
   "metadata": {},
   "source": [
    "We now use `DecisionTreeClassifier()`  to fit a classification tree in\n",
    "order to predict `High` using all variables but `Sales`.\n",
    "To do so, we must form a model matrix as we did when fitting regression\n",
    "models.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9fc6c8b9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:44.950122Z",
     "iopub.status.busy": "2023-08-21T02:29:44.950017Z",
     "iopub.status.idle": "2023-08-21T02:29:44.964042Z",
     "shell.execute_reply": "2023-08-21T02:29:44.963794Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "model = MS(Carseats.columns.drop('Sales'), intercept=False)\n",
    "D = model.fit_transform(Carseats)\n",
    "feature_names = list(D.columns)\n",
    "X = np.asarray(D)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86912d40",
   "metadata": {},
   "source": [
    "We have converted `D` from a data frame to an array `X`, which is needed in some of the analysis below. We also need the `feature_names` for annotating our plots later.\n",
    "\n",
    "There are several options needed to specify the  classifier,\n",
    "such as `max_depth` (how deep to grow the tree), `min_samples_split`\n",
    "(minimum number of observations in a node to be eligible for splitting)\n",
    "and `criterion` (whether to use Gini or cross-entropy as the split criterion).\n",
    "We also set `random_state` for reproducibility; ties in the split criterion are broken at random."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "533f0949",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:44.965639Z",
     "iopub.status.busy": "2023-08-21T02:29:44.965555Z",
     "iopub.status.idle": "2023-08-21T02:29:44.972079Z",
     "shell.execute_reply": "2023-08-21T02:29:44.971812Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>DecisionTreeClassifier(criterion=&#x27;entropy&#x27;, max_depth=3, random_state=0)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">DecisionTreeClassifier</label><div class=\"sk-toggleable__content\"><pre>DecisionTreeClassifier(criterion=&#x27;entropy&#x27;, max_depth=3, random_state=0)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "DecisionTreeClassifier(criterion='entropy', max_depth=3, random_state=0)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = DTC(criterion='entropy',\n",
    "          max_depth=3,\n",
    "          random_state=0)        \n",
    "clf.fit(X, High)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "118bae59",
   "metadata": {},
   "source": [
    "In our discussion of qualitative features in Section 3.3,\n",
    "we noted that for a linear regression model such a feature could be\n",
    "represented by including a matrix of dummy variables (one-hot-encoding) in the model\n",
    "matrix, using the formula notation of `statsmodels`.\n",
    "As mentioned in Section 8.1, there is a more\n",
    "natural way to handle qualitative features when building a decision\n",
    "tree, that does not require such dummy variables; each split amounts to partitioning the levels into two groups.\n",
    "However, \n",
    "the `sklearn` implementation of decision trees does not take\n",
    "advantage of this approach; instead it simply treats the one-hot-encoded levels as separate variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c4a8718e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:44.973649Z",
     "iopub.status.busy": "2023-08-21T02:29:44.973541Z",
     "iopub.status.idle": "2023-08-21T02:29:44.976459Z",
     "shell.execute_reply": "2023-08-21T02:29:44.976196Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.79"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(High, clf.predict(X))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcbbe99a",
   "metadata": {},
   "source": [
    "With only the default arguments, the training error rate is\n",
    "21%.\n",
    "For classification trees, we can\n",
    "access the value of the deviance using `log_loss()`,\n",
    "\\begin{equation*}\n",
    "\\begin{split}\n",
    "-2 \\sum_m \\sum_k n_{mk} \\log \\hat{p}_{mk},\n",
    "\\end{split}\n",
    "\\end{equation*}\n",
    "where $n_{mk}$ is the number of observations in the $m$th terminal\n",
    "node that belong to the $k$th class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2fe92cb1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:44.977909Z",
     "iopub.status.busy": "2023-08-21T02:29:44.977813Z",
     "iopub.status.idle": "2023-08-21T02:29:44.980904Z",
     "shell.execute_reply": "2023-08-21T02:29:44.980648Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4710647062649358"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resid_dev = np.sum(log_loss(High, clf.predict_proba(X)))\n",
    "resid_dev\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e47c068c",
   "metadata": {},
   "source": [
    "This is closely related to the *entropy*, defined in (8.7).\n",
    "A small deviance indicates a\n",
    "tree that provides a good fit to the (training) data.\n",
    " \n",
    "One of the most attractive properties of trees is that they can\n",
    "be graphically displayed. Here we use the `plot()`  function\n",
    "to display the tree structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "823745dc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:44.982460Z",
     "iopub.status.busy": "2023-08-21T02:29:44.982347Z",
     "iopub.status.idle": "2023-08-21T02:29:45.271601Z",
     "shell.execute_reply": "2023-08-21T02:29:45.271198Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 79044 characters]",
      "text/plain": [
       "<Figure size 1200x1200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = subplots(figsize=(12,12))[1]\n",
    "plot_tree(clf,\n",
    "          feature_names=feature_names,\n",
    "          ax=ax);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d089b674",
   "metadata": {},
   "source": [
    "The most important indicator of `Sales` appears to be `ShelveLoc`.\n",
    "\n",
    "We can see a text representation of the tree using\n",
    "`export_text()`, which displays the split\n",
    "criterion (e.g. `Price <= 92.5`) for each branch.\n",
    "For leaf nodes it shows the overall prediction  \n",
    "(`Yes` or `No`). \n",
    " We can also see the number of observations in that\n",
    "leaf that take on values of `Yes` and `No` by specifying  `show_weights=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "38ec5f13",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:45.273437Z",
     "iopub.status.busy": "2023-08-21T02:29:45.273325Z",
     "iopub.status.idle": "2023-08-21T02:29:45.276242Z",
     "shell.execute_reply": "2023-08-21T02:29:45.275887Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|--- ShelveLoc[Good] <= 0.50\n",
      "|   |--- Price <= 92.50\n",
      "|   |   |--- Income <= 57.00\n",
      "|   |   |   |--- weights: [7.00, 3.00] class: No\n",
      "|   |   |--- Income >  57.00\n",
      "|   |   |   |--- weights: [7.00, 29.00] class: Yes\n",
      "|   |--- Price >  92.50\n",
      "|   |   |--- Advertising <= 13.50\n",
      "|   |   |   |--- weights: [183.00, 41.00] class: No\n",
      "|   |   |--- Advertising >  13.50\n",
      "|   |   |   |--- weights: [20.00, 25.00] class: Yes\n",
      "|--- ShelveLoc[Good] >  0.50\n",
      "|   |--- Price <= 135.00\n",
      "|   |   |--- US[Yes] <= 0.50\n",
      "|   |   |   |--- weights: [6.00, 11.00] class: Yes\n",
      "|   |   |--- US[Yes] >  0.50\n",
      "|   |   |   |--- weights: [2.00, 49.00] class: Yes\n",
      "|   |--- Price >  135.00\n",
      "|   |   |--- Income <= 46.00\n",
      "|   |   |   |--- weights: [6.00, 0.00] class: No\n",
      "|   |   |--- Income >  46.00\n",
      "|   |   |   |--- weights: [5.00, 6.00] class: Yes\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(export_text(clf,\n",
    "                  feature_names=feature_names,\n",
    "                  show_weights=True))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f6a131b",
   "metadata": {},
   "source": [
    "In order to properly evaluate the performance of a classification tree\n",
    "on these data, we must estimate the test error rather than simply\n",
    "computing the training error. We split the observations into a\n",
    "training set and a test set, build the tree using the training set,\n",
    "and evaluate its performance on the test data. This pattern is\n",
    "similar to that in Chapter 6, with the linear models\n",
    "replaced here by decision trees --- the code for validation\n",
    "is almost identical. This approach leads to correct predictions\n",
    "for 68.5% of the locations in the test data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3959f39a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:45.277931Z",
     "iopub.status.busy": "2023-08-21T02:29:45.277817Z",
     "iopub.status.idle": "2023-08-21T02:29:45.284043Z",
     "shell.execute_reply": "2023-08-21T02:29:45.283649Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.685])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validation = skm.ShuffleSplit(n_splits=1,\n",
    "                              test_size=200,\n",
    "                              random_state=0)\n",
    "results = skm.cross_validate(clf,\n",
    "                             D,\n",
    "                             High,\n",
    "                             cv=validation)\n",
    "results['test_score']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e226364",
   "metadata": {},
   "source": [
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "710f158a",
   "metadata": {},
   "source": [
    "Next, we consider whether pruning the tree might lead to improved\n",
    "classification performance. We first split the data into a training and\n",
    "test set. We will use cross-validation to prune the tree on the training\n",
    "set, and then evaluate the performance of the pruned tree on the test\n",
    "set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "201c4690",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:45.286011Z",
     "iopub.status.busy": "2023-08-21T02:29:45.285866Z",
     "iopub.status.idle": "2023-08-21T02:29:45.288396Z",
     "shell.execute_reply": "2023-08-21T02:29:45.288038Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "(X_train,\n",
    " X_test,\n",
    " High_train,\n",
    " High_test) = skm.train_test_split(X,\n",
    "                                   High,\n",
    "                                   test_size=0.5,\n",
    "                                   random_state=0)\n",
    "                                   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eed4457",
   "metadata": {},
   "source": [
    "We first refit the full tree on the training set; here we do not set a `max_depth` parameter, since we will learn that through cross-validation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a8dc5c3a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:45.290090Z",
     "iopub.status.busy": "2023-08-21T02:29:45.289979Z",
     "iopub.status.idle": "2023-08-21T02:29:45.293957Z",
     "shell.execute_reply": "2023-08-21T02:29:45.293696Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.735"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = DTC(criterion='entropy', random_state=0)\n",
    "clf.fit(X_train, High_train)\n",
    "accuracy_score(High_test, clf.predict(X_test))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78654680",
   "metadata": {},
   "source": [
    "Next we use the `cost_complexity_pruning_path()` method of\n",
    "`clf` to extract cost-complexity values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0cbe0d28",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:45.295541Z",
     "iopub.status.busy": "2023-08-21T02:29:45.295458Z",
     "iopub.status.idle": "2023-08-21T02:29:45.299182Z",
     "shell.execute_reply": "2023-08-21T02:29:45.298807Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "ccp_path = clf.cost_complexity_pruning_path(X_train, High_train)\n",
    "kfold = skm.KFold(10,\n",
    "                  random_state=1,\n",
    "                  shuffle=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "854bd2fe",
   "metadata": {},
   "source": [
    "This yields a set of impurities and $\\alpha$ values\n",
    "from which we can extract an optimal one by cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ea649080",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:45.300868Z",
     "iopub.status.busy": "2023-08-21T02:29:45.300771Z",
     "iopub.status.idle": "2023-08-21T02:29:45.571746Z",
     "shell.execute_reply": "2023-08-21T02:29:45.571364Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.685"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid = skm.GridSearchCV(clf,\n",
    "                        {'ccp_alpha': ccp_path.ccp_alphas},\n",
    "                        refit=True,\n",
    "                        cv=kfold,\n",
    "                        scoring='accuracy')\n",
    "grid.fit(X_train, High_train)\n",
    "grid.best_score_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b8e7965",
   "metadata": {},
   "source": [
    "Let’s take a look at the pruned true."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e005da14",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:45.573701Z",
     "iopub.status.busy": "2023-08-21T02:29:45.573563Z",
     "iopub.status.idle": "2023-08-21T02:29:46.379024Z",
     "shell.execute_reply": "2023-08-21T02:29:46.378601Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 127612 characters]",
      "text/plain": [
       "<Figure size 1200x1200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = subplots(figsize=(12, 12))[1]\n",
    "best_ = grid.best_estimator_\n",
    "plot_tree(best_,\n",
    "          feature_names=feature_names,\n",
    "          ax=ax);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "923e44d2",
   "metadata": {},
   "source": [
    "This is quite a bushy tree. We could count the leaves, or query\n",
    "`best_` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "23324f7e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:46.381085Z",
     "iopub.status.busy": "2023-08-21T02:29:46.380960Z",
     "iopub.status.idle": "2023-08-21T02:29:46.383318Z",
     "shell.execute_reply": "2023-08-21T02:29:46.383057Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_.tree_.n_leaves\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0d9bb88",
   "metadata": {},
   "source": [
    "The tree with 30 terminal\n",
    "nodes results in the lowest cross-validation error rate, with an accuracy of\n",
    "68.5%. How well does this pruned tree perform on the test data set? Once\n",
    "again, we apply the `predict()`  function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6dcd8b37",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:46.385077Z",
     "iopub.status.busy": "2023-08-21T02:29:46.384936Z",
     "iopub.status.idle": "2023-08-21T02:29:46.392841Z",
     "shell.execute_reply": "2023-08-21T02:29:46.392541Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.72\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Truth</th>\n",
       "      <th>No</th>\n",
       "      <th>Yes</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Predicted</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>No</th>\n",
       "      <td>94</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>24</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Truth      No  Yes\n",
       "Predicted         \n",
       "No         94   32\n",
       "Yes        24   50"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(accuracy_score(High_test,\n",
    "                     best_.predict(X_test)))\n",
    "confusion = confusion_table(best_.predict(X_test),\n",
    "                            High_test)\n",
    "confusion\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42c39ca1",
   "metadata": {},
   "source": [
    "Now 72.0% of the test observations are correctly classified, which is slightly worse than the error for the full tree (with 35 leaves). So cross-validation has not helped us much here; it only pruned off 5 leaves, at a cost of a slightly worse error. These results would change if we were to change the random number seeds above; even though cross-validation gives an unbiased approach to model selection, it does have variance.\n",
    "\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94ac4462",
   "metadata": {},
   "source": [
    "## Fitting Regression Trees\n",
    "Here we fit a regression tree to the  `Boston`  data set. The\n",
    "steps are similar to those for classification trees."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0459d3a9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:46.395016Z",
     "iopub.status.busy": "2023-08-21T02:29:46.394693Z",
     "iopub.status.idle": "2023-08-21T02:29:46.408492Z",
     "shell.execute_reply": "2023-08-21T02:29:46.408187Z"
    }
   },
   "outputs": [],
   "source": [
    "Boston = load_data(\"Boston\")\n",
    "model = MS(Boston.columns.drop('medv'), intercept=False)\n",
    "D = model.fit_transform(Boston)\n",
    "feature_names = list(D.columns)\n",
    "X = np.asarray(D)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b09cc1f",
   "metadata": {},
   "source": [
    "First, we split the data into training and test sets, and fit the tree\n",
    "to the training data. Here we use 30% of the data for the test set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "34bf2864",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:46.411065Z",
     "iopub.status.busy": "2023-08-21T02:29:46.410850Z",
     "iopub.status.idle": "2023-08-21T02:29:46.413541Z",
     "shell.execute_reply": "2023-08-21T02:29:46.413268Z"
    }
   },
   "outputs": [],
   "source": [
    "(X_train,\n",
    " X_test,\n",
    " y_train,\n",
    " y_test) = skm.train_test_split(X,\n",
    "                                Boston['medv'],\n",
    "                                test_size=0.3,\n",
    "                                random_state=0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc5ddbb2",
   "metadata": {},
   "source": [
    "Having formed  our training  and test data sets, we fit the regression tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "dd0dfd8a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:46.415108Z",
     "iopub.status.busy": "2023-08-21T02:29:46.414996Z",
     "iopub.status.idle": "2023-08-21T02:29:46.704739Z",
     "shell.execute_reply": "2023-08-21T02:29:46.704318Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 83364 characters]",
      "text/plain": [
       "<Figure size 1200x1200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "reg = DTR(max_depth=3)\n",
    "reg.fit(X_train, y_train)\n",
    "ax = subplots(figsize=(12,12))[1]\n",
    "plot_tree(reg,\n",
    "          feature_names=feature_names,\n",
    "          ax=ax);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62207ca5",
   "metadata": {},
   "source": [
    "The variable `lstat` measures the percentage of individuals with\n",
    "lower socioeconomic status. The tree indicates that lower\n",
    "values of `lstat` correspond to more expensive houses.\n",
    "The tree predicts a median house price of $12,042 for small-sized homes (`rm < 6.8`), in\n",
    "suburbs in which residents have low socioeconomic status (`lstat  > 14.4`) and the crime-rate is moderate (`crim > 5.8`)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "361d7f84",
   "metadata": {},
   "source": [
    "Now we use the cross-validation function to see whether pruning\n",
    "the tree will improve performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "33fb7786",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:46.706833Z",
     "iopub.status.busy": "2023-08-21T02:29:46.706699Z",
     "iopub.status.idle": "2023-08-21T02:29:46.756824Z",
     "shell.execute_reply": "2023-08-21T02:29:46.756520Z"
    }
   },
   "outputs": [],
   "source": [
    "ccp_path = reg.cost_complexity_pruning_path(X_train, y_train)\n",
    "kfold = skm.KFold(5,\n",
    "                  shuffle=True,\n",
    "                  random_state=10)\n",
    "grid = skm.GridSearchCV(reg,\n",
    "                        {'ccp_alpha': ccp_path.ccp_alphas},\n",
    "                        refit=True,\n",
    "                        cv=kfold,\n",
    "                        scoring='neg_mean_squared_error')\n",
    "G = grid.fit(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7218495",
   "metadata": {},
   "source": [
    "In keeping with the cross-validation results, we use the pruned tree\n",
    "to make predictions on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a386755b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:46.758663Z",
     "iopub.status.busy": "2023-08-21T02:29:46.758546Z",
     "iopub.status.idle": "2023-08-21T02:29:46.761814Z",
     "shell.execute_reply": "2023-08-21T02:29:46.761470Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.06985754975404"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_ = grid.best_estimator_\n",
    "np.mean((y_test - best_.predict(X_test))**2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fe01e33",
   "metadata": {},
   "source": [
    "In other words, the test set MSE associated with the regression tree\n",
    "is 28.07.  The square root of\n",
    "the MSE is therefore around\n",
    "5.30,\n",
    "indicating that this model leads to test predictions that are within around\n",
    "$5300\n",
    "of the true median home value for the suburb.\n",
    "\n",
    "Let’s plot the best tree to see how interpretable it is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3d646928",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:46.763419Z",
     "iopub.status.busy": "2023-08-21T02:29:46.763331Z",
     "iopub.status.idle": "2023-08-21T02:29:47.049615Z",
     "shell.execute_reply": "2023-08-21T02:29:47.049265Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 83368 characters]",
      "text/plain": [
       "<Figure size 1200x1200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = subplots(figsize=(12,12))[1]\n",
    "plot_tree(G.best_estimator_,\n",
    "          feature_names=feature_names,\n",
    "          ax=ax);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "074e230c",
   "metadata": {},
   "source": [
    " \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa0e4849",
   "metadata": {},
   "source": [
    "## Bagging and Random Forests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b17ca840",
   "metadata": {},
   "source": [
    "Here we apply bagging and random forests to the `Boston` data, using\n",
    "the `RandomForestRegressor()` from the `sklearn.ensemble` package. Recall\n",
    "that bagging is simply a special case of a random forest with\n",
    "$m=p$. Therefore, the `RandomForestRegressor()`  function can be used to\n",
    "perform both bagging and random forests. We start with bagging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "275feef7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:47.051447Z",
     "iopub.status.busy": "2023-08-21T02:29:47.051323Z",
     "iopub.status.idle": "2023-08-21T02:29:47.211049Z",
     "shell.execute_reply": "2023-08-21T02:29:47.210740Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {color: black;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>RandomForestRegressor(max_features=12, random_state=0)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestRegressor</label><div class=\"sk-toggleable__content\"><pre>RandomForestRegressor(max_features=12, random_state=0)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "RandomForestRegressor(max_features=12, random_state=0)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bag_boston = RF(max_features=X_train.shape[1], random_state=0)\n",
    "bag_boston.fit(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e84da6",
   "metadata": {},
   "source": [
    "The argument `max_features` indicates that all 12 predictors should\n",
    "be considered for each split of the tree --- in other words, that\n",
    "bagging should be done.  How well does this bagged model perform on\n",
    "the test set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "01dbbef3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:47.212716Z",
     "iopub.status.busy": "2023-08-21T02:29:47.212598Z",
     "iopub.status.idle": "2023-08-21T02:29:47.305563Z",
     "shell.execute_reply": "2023-08-21T02:29:47.305155Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14.634700151315787"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 26324 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = subplots(figsize=(8,8))[1]\n",
    "y_hat_bag = bag_boston.predict(X_test)\n",
    "ax.scatter(y_hat_bag, y_test)\n",
    "np.mean((y_test - y_hat_bag)**2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "885a3843",
   "metadata": {},
   "source": [
    "The test set MSE associated with the bagged regression tree is\n",
    "14.63, about half that obtained using an optimally-pruned single\n",
    "tree.  We could change the number of trees grown from the default of\n",
    "100 by\n",
    "using the `n_estimators` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b75cc90e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:47.307949Z",
     "iopub.status.busy": "2023-08-21T02:29:47.307809Z",
     "iopub.status.idle": "2023-08-21T02:29:48.296516Z",
     "shell.execute_reply": "2023-08-21T02:29:48.296211Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14.605662565263161"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bag_boston = RF(max_features=X_train.shape[1],\n",
    "                n_estimators=500,\n",
    "                random_state=0).fit(X_train, y_train)\n",
    "y_hat_bag = bag_boston.predict(X_test)\n",
    "np.mean((y_test - y_hat_bag)**2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37d5fe61",
   "metadata": {},
   "source": [
    "There is not much change. Bagging and random forests cannot overfit by\n",
    "increasing the number of trees, but can underfit if the number is too small.\n",
    "\n",
    "Growing a random forest proceeds in exactly the same way, except that\n",
    "we use a smaller value of the `max_features` argument. By default,\n",
    "`RandomForestRegressor()`  uses $p$ variables when building a random\n",
    "forest of regression trees (i.e. it defaults to bagging), and `RandomForestClassifier()` uses\n",
    "$\\sqrt{p}$ variables when building a\n",
    "random forest of classification trees. Here we use `max_features=6`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "bf9a5ed4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:48.298333Z",
     "iopub.status.busy": "2023-08-21T02:29:48.298184Z",
     "iopub.status.idle": "2023-08-21T02:29:48.412949Z",
     "shell.execute_reply": "2023-08-21T02:29:48.412594Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20.04276446710527"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "RF_boston = RF(max_features=6,\n",
    "               random_state=0).fit(X_train, y_train)\n",
    "y_hat_RF = RF_boston.predict(X_test)\n",
    "np.mean((y_test - y_hat_RF)**2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb2c3917",
   "metadata": {},
   "source": [
    "The test set MSE is 20.04;\n",
    "this indicates that random forests did somewhat worse than bagging\n",
    "in this case. Extracting the `feature_importances_` values from the fitted model, we can view the\n",
    "importance of each variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "71316e9a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:48.414933Z",
     "iopub.status.busy": "2023-08-21T02:29:48.414785Z",
     "iopub.status.idle": "2023-08-21T02:29:48.422251Z",
     "shell.execute_reply": "2023-08-21T02:29:48.421922Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>lstat</th>\n",
       "      <td>0.356203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rm</th>\n",
       "      <td>0.332163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ptratio</th>\n",
       "      <td>0.067270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>crim</th>\n",
       "      <td>0.055404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>indus</th>\n",
       "      <td>0.053851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dis</th>\n",
       "      <td>0.041582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nox</th>\n",
       "      <td>0.035225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tax</th>\n",
       "      <td>0.025355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>0.021506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rad</th>\n",
       "      <td>0.004784</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chas</th>\n",
       "      <td>0.004203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zn</th>\n",
       "      <td>0.002454</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         importance\n",
       "lstat      0.356203\n",
       "rm         0.332163\n",
       "ptratio    0.067270\n",
       "crim       0.055404\n",
       "indus      0.053851\n",
       "dis        0.041582\n",
       "nox        0.035225\n",
       "tax        0.025355\n",
       "age        0.021506\n",
       "rad        0.004784\n",
       "chas       0.004203\n",
       "zn         0.002454"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_imp = pd.DataFrame(\n",
    "    {'importance':RF_boston.feature_importances_},\n",
    "    index=feature_names)\n",
    "feature_imp.sort_values(by='importance', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3cd6a28",
   "metadata": {},
   "source": [
    " This\n",
    "is a relative measure of the total decrease in node impurity that results from\n",
    "splits over that variable, averaged over all trees (this was plotted in Figure 8.9 for a model fit to the `Heart` data). \n",
    "\n",
    "The results indicate that across all of the trees considered in the\n",
    "random forest, the wealth level of the community (`lstat`) and the\n",
    "house size (`rm`) are by far the two most important variables.\n",
    "\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "270fc0fb",
   "metadata": {},
   "source": [
    "## Boosting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bd28754",
   "metadata": {},
   "source": [
    "Here we use `GradientBoostingRegressor()` from `sklearn.ensemble`\n",
    "to fit boosted regression trees to the `Boston` data\n",
    "set. For classification we would  use `GradientBoostingClassifier()`.\n",
    "The argument `n_estimators=5000`\n",
    "indicates that we want 5000 trees, and the option\n",
    "`max_depth=3` limits the depth of each tree. The\n",
    "argument `learning_rate` is the $\\lambda$\n",
    "mentioned earlier in the description of boosting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "0bcc5ff1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:48.424109Z",
     "iopub.status.busy": "2023-08-21T02:29:48.423996Z",
     "iopub.status.idle": "2023-08-21T02:29:52.311829Z",
     "shell.execute_reply": "2023-08-21T02:29:52.311492Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-3 {color: black;}#sk-container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-3 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align: center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GradientBoostingRegressor(learning_rate=0.001, n_estimators=5000,\n",
       "                          random_state=0)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" checked><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GradientBoostingRegressor</label><div class=\"sk-toggleable__content\"><pre>GradientBoostingRegressor(learning_rate=0.001, n_estimators=5000,\n",
       "                          random_state=0)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "GradientBoostingRegressor(learning_rate=0.001, n_estimators=5000,\n",
       "                          random_state=0)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "boost_boston = GBR(n_estimators=5000,\n",
    "                   learning_rate=0.001,\n",
    "                   max_depth=3,\n",
    "                   random_state=0)\n",
    "boost_boston.fit(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e8553ec",
   "metadata": {},
   "source": [
    "We can see how the training error decreases with the `train_score_` attribute.\n",
    "To get an idea of how the test error decreases we can use the\n",
    "`staged_predict()` method to get the predicted values along the path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "060f47eb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:52.313643Z",
     "iopub.status.busy": "2023-08-21T02:29:52.313522Z",
     "iopub.status.idle": "2023-08-21T02:29:52.788145Z",
     "shell.execute_reply": "2023-08-21T02:29:52.787823Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 32720 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_error = np.zeros_like(boost_boston.train_score_)\n",
    "for idx, y_ in enumerate(boost_boston.staged_predict(X_test)):\n",
    "   test_error[idx] = np.mean((y_test - y_)**2)\n",
    "\n",
    "plot_idx = np.arange(boost_boston.train_score_.shape[0])\n",
    "ax = subplots(figsize=(8,8))[1]\n",
    "ax.plot(plot_idx,\n",
    "        boost_boston.train_score_,\n",
    "        'b',\n",
    "        label='Training')\n",
    "ax.plot(plot_idx,\n",
    "        test_error,\n",
    "        'r',\n",
    "        label='Test')\n",
    "ax.legend();\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ebd0944",
   "metadata": {},
   "source": [
    "We now use the boosted model to predict `medv` on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "43505dad",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:52.789753Z",
     "iopub.status.busy": "2023-08-21T02:29:52.789632Z",
     "iopub.status.idle": "2023-08-21T02:29:52.801529Z",
     "shell.execute_reply": "2023-08-21T02:29:52.801234Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14.481405918831591"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_hat_boost = boost_boston.predict(X_test);\n",
    "np.mean((y_test - y_hat_boost)**2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f76bb89",
   "metadata": {},
   "source": [
    " The test MSE obtained is 14.48,\n",
    "similar to the test MSE for bagging. If we want to, we can\n",
    "perform boosting with a different value of the shrinkage parameter\n",
    "$\\lambda$ in  (8.10). The default value is 0.001, but\n",
    "this is easily modified.  Here we take $\\lambda=0.2$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "c0a03126",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:52.803315Z",
     "iopub.status.busy": "2023-08-21T02:29:52.803174Z",
     "iopub.status.idle": "2023-08-21T02:29:55.330726Z",
     "shell.execute_reply": "2023-08-21T02:29:55.330425Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14.501514553719565"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "boost_boston = GBR(n_estimators=5000,\n",
    "                   learning_rate=0.2,\n",
    "                   max_depth=3,\n",
    "                   random_state=0)\n",
    "boost_boston.fit(X_train,\n",
    "                 y_train)\n",
    "y_hat_boost = boost_boston.predict(X_test);\n",
    "np.mean((y_test - y_hat_boost)**2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbcad91c",
   "metadata": {},
   "source": [
    "In this case, using $\\lambda=0.2$ leads to a almost the same test MSE\n",
    "as when using $\\lambda=0.001$.\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d561719",
   "metadata": {},
   "source": [
    "## Bayesian Additive Regression Trees"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f68c0d8f",
   "metadata": {},
   "source": [
    "In this section we demonstrate a  `Python` implementation of BART found in the\n",
    "`ISLP.bart` package. We fit a  model\n",
    "to the `Boston` housing data set. This `BART()` estimator is\n",
    "designed for quantitative outcome variables, though other implementations are available for\n",
    "fitting logistic and probit models to categorical outcomes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "58f6e11f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:55.332677Z",
     "iopub.status.busy": "2023-08-21T02:29:55.332550Z",
     "iopub.status.idle": "2023-08-21T02:29:56.797204Z",
     "shell.execute_reply": "2023-08-21T02:29:56.796899Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-4 {color: black;}#sk-container-id-4 pre{padding: 0;}#sk-container-id-4 div.sk-toggleable {background-color: white;}#sk-container-id-4 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-4 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-4 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-4 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-4 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-4 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-4 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-4 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-4 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-4 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-4 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-4 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-4 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-4 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-4 div.sk-item {position: relative;z-index: 1;}#sk-container-id-4 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-4 div.sk-item::before, #sk-container-id-4 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-4 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-4 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-4 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-4 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-4 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-4 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-4 div.sk-label-container {text-align: center;}#sk-container-id-4 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-4 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-4\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>BART(burnin=5, ndraw=15, random_state=0)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" checked><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">BART</label><div class=\"sk-toggleable__content\"><pre>BART(burnin=5, ndraw=15, random_state=0)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "BART(burnin=5, ndraw=15, random_state=0)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bart_boston = BART(random_state=0, burnin=5, ndraw=15)\n",
    "bart_boston.fit(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26e4d574",
   "metadata": {},
   "source": [
    "On this data set, with this split into test and training, we see that the test error of BART is similar to that of  random forest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "a5b1296f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:56.798892Z",
     "iopub.status.busy": "2023-08-21T02:29:56.798779Z",
     "iopub.status.idle": "2023-08-21T02:29:57.098494Z",
     "shell.execute_reply": "2023-08-21T02:29:57.098140Z"
    },
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20.739185417498764"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yhat_test = bart_boston.predict(X_test.astype(np.float32))\n",
    "np.mean((y_test - yhat_test)**2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92a60f97",
   "metadata": {},
   "source": [
    "We can check how many times each variable appeared in the collection of trees.\n",
    "This gives a summary similar to the variable importance plot for boosting and random forests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "36c52755",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-21T02:29:57.100444Z",
     "iopub.status.busy": "2023-08-21T02:29:57.100310Z",
     "iopub.status.idle": "2023-08-21T02:29:57.103577Z",
     "shell.execute_reply": "2023-08-21T02:29:57.103213Z"
    },
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "crim       25.466667\n",
       "zn         30.600000\n",
       "indus      24.933333\n",
       "chas       21.133333\n",
       "nox        27.333333\n",
       "rm         28.800000\n",
       "age        23.466667\n",
       "dis        26.000000\n",
       "rad        25.000000\n",
       "tax        21.733333\n",
       "ptratio    26.800000\n",
       "lstat      31.866667\n",
       "dtype: float64"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var_inclusion = pd.Series(bart_boston.variable_inclusion_.mean(0),\n",
    "                               index=D.columns)\n",
    "var_inclusion\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab69e5b5",
   "metadata": {},
   "source": [
    "    \n",
    "  \n",
    " \n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb,Rmd",
   "main_language": "python"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
