v2.1 of Ch13

Ch13-multiple-lab.Rmd

@@ -1,16 +1,3 @@
----
-jupyter:
-  jupytext:
-    cell_metadata_filter: -all
-    formats: ipynb,Rmd
-    main_language: python
-    text_representation:
-      extension: .Rmd
-      format_name: rmarkdown
-      format_version: '1.2'
-      jupytext_version: 1.14.7
----
-
 
 # Chapter 13
 
@@ -245,6 +232,7 @@ mult_test(fund_mini_pvals, method = "holm", alpha=0.05)[:2]
 As discussed previously, Manager One seems to perform particularly
 well, whereas Manager Two has poor performance.
  
+
 ```{python}
 fund_mini.mean()
 

Ch13-multiple-lab.ipynb

@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "75b2d75c",
+   "id": "687e9b80",
    "metadata": {},
    "source": [
     "\n",
@@ -14,7 +14,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "34e410a6",
+   "id": "9f594a41",
    "metadata": {},
    "source": [
     "We include our usual imports seen in earlier labs."
@@ -23,13 +23,13 @@
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "1f928b2d",
+   "id": "7cc4fbeb",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:48.489971Z",
-     "iopub.status.busy": "2023-08-07T00:19:48.489727Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.216508Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.215573Z"
+     "iopub.execute_input": "2023-08-21T02:29:16.417394Z",
+     "iopub.status.busy": "2023-08-21T02:29:16.417287Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.613483Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.613156Z"
     }
    },
    "outputs": [],
@@ -43,7 +43,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12319e0a",
+   "id": "08ba7bed",
    "metadata": {},
    "source": [
     "We also collect the new imports\n",
@@ -53,13 +53,13 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "eb4b32aa",
+   "id": "595efc18",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.219989Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.219429Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.223239Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.222392Z"
+     "iopub.execute_input": "2023-08-21T02:29:17.615551Z",
+     "iopub.status.busy": "2023-08-21T02:29:17.615375Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.617379Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.617087Z"
     },
     "lines_to_next_cell": 2
    },
@@ -78,7 +78,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a2747e58",
+   "id": "69e5023e",
    "metadata": {},
    "source": [
     "## Review of Hypothesis Tests\n",
@@ -92,13 +92,13 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "e12ac0cd",
+   "id": "985d1d6e",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.225454Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.225335Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.228651Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.228301Z"
+     "iopub.execute_input": "2023-08-21T02:29:17.618995Z",
+     "iopub.status.busy": "2023-08-21T02:29:17.618887Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.620921Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.620629Z"
     }
    },
    "outputs": [],
@@ -111,7 +111,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "70d37233",
+   "id": "9ae561c4",
    "metadata": {},
    "source": [
     "To begin, we use `ttest_1samp()`  from the\n",
@@ -122,13 +122,13 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "04d0f49e",
+   "id": "753d612a",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.230854Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.230727Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.236745Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.236388Z"
+     "iopub.execute_input": "2023-08-21T02:29:17.622537Z",
+     "iopub.status.busy": "2023-08-21T02:29:17.622429Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.626063Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.625801Z"
     }
    },
    "outputs": [
@@ -150,7 +150,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cf83426f",
+   "id": "5d9dc17f",
    "metadata": {},
    "source": [
     "The $p$-value comes out to 0.931, which is not low enough to\n",
@@ -169,13 +169,13 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "id": "d1f0c695",
+   "id": "facd6569",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.238822Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.238685Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.263876Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.263508Z"
+     "iopub.execute_input": "2023-08-21T02:29:17.627714Z",
+     "iopub.status.busy": "2023-08-21T02:29:17.627617Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.651726Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.651448Z"
     },
     "lines_to_next_cell": 0
    },
@@ -195,7 +195,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3d8e0d96",
+   "id": "4094daa7",
    "metadata": {},
    "source": [
     "Since this is a simulated data set, we can create a $2 \\times 2$ table\n",
@@ -205,13 +205,13 @@
   {
    "cell_type": "code",
    "execution_count": 6,
-   "id": "7a9594a0",
+   "id": "e89ef3eb",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.266708Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.266387Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.280831Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.280194Z"
+     "iopub.execute_input": "2023-08-21T02:29:17.653344Z",
+     "iopub.status.busy": "2023-08-21T02:29:17.653256Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.662644Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.662346Z"
     },
     "lines_to_next_cell": 0
    },
@@ -282,7 +282,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9610c817",
+   "id": "a10ba423",
    "metadata": {},
    "source": [
     "Therefore, at level $\\alpha=0.05$, we reject 15 of the 50 false\n",
@@ -299,19 +299,20 @@
     "amounts to quite a weak signal, and it resulted in a high number of\n",
     "Type II errors.  Let’s instead simulate data with a stronger signal,\n",
     "so that the ratio of the mean to the standard deviation for the false\n",
-    "null hypotheses equals $1$. We make only 10 Type II errors.\n"
+    "null hypotheses equals $1$. We make only 10 Type II errors.\n",
+    " "
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 7,
-   "id": "25f7fc5d",
+   "id": "ae184aaf",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.283336Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.283198Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.317664Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.317356Z"
+     "iopub.execute_input": "2023-08-21T02:29:17.664327Z",
+     "iopub.status.busy": "2023-08-21T02:29:17.664213Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.690928Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.690657Z"
     },
     "lines_to_next_cell": 0
    },
@@ -394,7 +395,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bb70c597",
+   "id": "7ca15d3f",
    "metadata": {},
    "source": [
     "    "
@@ -402,7 +403,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f6953d33",
+   "id": "9e9f5977",
    "metadata": {},
    "source": [
     "## Family-Wise Error Rate\n",
@@ -417,13 +418,13 @@
   {
    "cell_type": "code",
    "execution_count": 8,
-   "id": "369b5bd3",
+   "id": "0295fe68",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.319399Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.319268Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.674219Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.673887Z"
+     "iopub.execute_input": "2023-08-21T02:29:17.692568Z",
+     "iopub.status.busy": "2023-08-21T02:29:17.692459Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.899403Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.899081Z"
     }
    },
    "outputs": [
@@ -454,7 +455,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3a81479e",
+   "id": "fecaca9e",
    "metadata": {},
    "source": [
     "As discussed previously, even for moderate values of $m$ such as $50$,\n",
@@ -473,13 +474,13 @@
   {
    "cell_type": "code",
    "execution_count": 9,
-   "id": "9ce7a19f",
+   "id": "406e59a8",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.676029Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.675909Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.715451Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.715145Z"
+     "iopub.execute_input": "2023-08-21T02:29:17.901146Z",
+     "iopub.status.busy": "2023-08-21T02:29:17.901041Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.939312Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.939019Z"
     }
    },
    "outputs": [
@@ -505,7 +506,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7561e3a3",
+   "id": "87bab88b",
    "metadata": {},
    "source": [
     "The $p$-values are low for Managers One and Three, and high for the\n",
@@ -530,7 +531,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5b608e46",
+   "id": "d0c7a2a0",
    "metadata": {},
    "source": [
     "The `mult_test()` function takes $p$-values and a `method` argument, as well as an optional\n",
@@ -541,13 +542,13 @@
   {
    "cell_type": "code",
    "execution_count": 10,
-   "id": "de6cffed",
+   "id": "d4f6a247",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.717476Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.717258Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.719841Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.719577Z"
+     "iopub.execute_input": "2023-08-21T02:29:17.941032Z",
+     "iopub.status.busy": "2023-08-21T02:29:17.940919Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.943369Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.943081Z"
     },
     "lines_to_next_cell": 2
    },
@@ -570,7 +571,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5135c6b9",
+   "id": "4d5bc7e7",
    "metadata": {},
    "source": [
     "The $p$-values `bonf` are simply the `fund_mini_pvalues` multiplied by 5 and truncated to be less than\n",
@@ -580,13 +581,13 @@
   {
    "cell_type": "code",
    "execution_count": 11,
-   "id": "0de71500",
+   "id": "01a29d71",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.721450Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.721342Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.723962Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.723691Z"
+     "iopub.execute_input": "2023-08-21T02:29:17.944859Z",
+     "iopub.status.busy": "2023-08-21T02:29:17.944760Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.946888Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.946639Z"
     }
    },
    "outputs": [
@@ -608,7 +609,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1f0bc112",
+   "id": "e68c9051",
    "metadata": {},
    "source": [
     "Therefore, using Bonferroni’s method, we are able to reject the null hypothesis only for Manager\n",
@@ -622,13 +623,13 @@
   {
    "cell_type": "code",
    "execution_count": 12,
-   "id": "f7e87bdb",
+   "id": "95454eb4",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.725450Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.725349Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.777055Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.776612Z"
+     "iopub.execute_input": "2023-08-21T02:29:17.948474Z",
+     "iopub.status.busy": "2023-08-21T02:29:17.948372Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.990740Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.990464Z"
     },
     "lines_to_next_cell": 2
    },
@@ -651,23 +652,24 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f762fecd",
+   "id": "587b5b48",
    "metadata": {},
    "source": [
     "As discussed previously, Manager One seems to perform particularly\n",
-    "well, whereas Manager Two has poor performance."
+    "well, whereas Manager Two has poor performance.\n",
+    " "
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 13,
-   "id": "e88be376",
+   "id": "1f1ac764",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.779763Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.779554Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.783798Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.783418Z"
+     "iopub.execute_input": "2023-08-21T02:29:17.992261Z",
+     "iopub.status.busy": "2023-08-21T02:29:17.992149Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.995141Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.994894Z"
     },
     "lines_to_next_cell": 2
    },
@@ -694,7 +696,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "88dbf0a6",
+   "id": "e3c4b716",
    "metadata": {},
    "source": [
     "Is there evidence of a meaningful difference in performance between\n",
@@ -705,13 +707,13 @@
   {
    "cell_type": "code",
    "execution_count": 14,
-   "id": "41149af6",
+   "id": "298d975d",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.786752Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.786580Z",
-     "iopub.status.idle": "2023-08-07T00:19:50.791095Z",
-     "shell.execute_reply": "2023-08-07T00:19:50.790607Z"
+     "iopub.execute_input": "2023-08-21T02:29:17.996686Z",
+     "iopub.status.busy": "2023-08-21T02:29:17.996590Z",
+     "iopub.status.idle": "2023-08-21T02:29:17.999332Z",
+     "shell.execute_reply": "2023-08-21T02:29:17.999076Z"
     }
    },
    "outputs": [
@@ -733,7 +735,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1aca6122",
+   "id": "3908d7d2",
    "metadata": {},
    "source": [
     "The test results in a $p$-value of 0.038,\n",
@@ -757,13 +759,13 @@
   {
    "cell_type": "code",
    "execution_count": 15,
-   "id": "61aabda7",
+   "id": "be117713",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:50.793577Z",
-     "iopub.status.busy": "2023-08-07T00:19:50.793394Z",
-     "iopub.status.idle": "2023-08-07T00:19:51.318999Z",
-     "shell.execute_reply": "2023-08-07T00:19:51.318071Z"
+     "iopub.execute_input": "2023-08-21T02:29:18.000853Z",
+     "iopub.status.busy": "2023-08-21T02:29:18.000747Z",
+     "iopub.status.idle": "2023-08-21T02:29:18.487357Z",
+     "shell.execute_reply": "2023-08-21T02:29:18.487078Z"
     },
     "lines_to_next_cell": 2
    },
@@ -799,7 +801,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e0084fc5",
+   "id": "0fdf963f",
    "metadata": {},
    "source": [
     "The `pairwise_tukeyhsd()` function provides confidence intervals\n",
@@ -817,13 +819,13 @@
   {
    "cell_type": "code",
    "execution_count": 16,
-   "id": "cbcad4de",
+   "id": "537c4ea8",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:51.321200Z",
-     "iopub.status.busy": "2023-08-07T00:19:51.321046Z",
-     "iopub.status.idle": "2023-08-07T00:19:51.411142Z",
-     "shell.execute_reply": "2023-08-07T00:19:51.410682Z"
+     "iopub.execute_input": "2023-08-21T02:29:18.489069Z",
+     "iopub.status.busy": "2023-08-21T02:29:18.488949Z",
+     "iopub.status.idle": "2023-08-21T02:29:18.570869Z",
+     "shell.execute_reply": "2023-08-21T02:29:18.570427Z"
     }
    },
    "outputs": [
@@ -845,7 +847,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6278d13c",
+   "id": "8590f246",
    "metadata": {},
    "source": [
     "## False Discovery Rate\n",
@@ -858,13 +860,13 @@
   {
    "cell_type": "code",
    "execution_count": 17,
-   "id": "b5842190",
+   "id": "2c88ec87",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:51.413331Z",
-     "iopub.status.busy": "2023-08-07T00:19:51.413176Z",
-     "iopub.status.idle": "2023-08-07T00:19:51.848427Z",
-     "shell.execute_reply": "2023-08-07T00:19:51.847956Z"
+     "iopub.execute_input": "2023-08-21T02:29:18.572454Z",
+     "iopub.status.busy": "2023-08-21T02:29:18.572341Z",
+     "iopub.status.idle": "2023-08-21T02:29:19.005707Z",
+     "shell.execute_reply": "2023-08-21T02:29:19.005387Z"
     }
    },
    "outputs": [],
@@ -876,7 +878,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "80fc2fcc",
+   "id": "80e77fab",
    "metadata": {},
    "source": [
     "There are far too many managers to consider trying to control the FWER.\n",
@@ -887,13 +889,13 @@
   {
    "cell_type": "code",
    "execution_count": 18,
-   "id": "7c9d8bed",
+   "id": "b6d56819",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:51.850663Z",
-     "iopub.status.busy": "2023-08-07T00:19:51.850523Z",
-     "iopub.status.idle": "2023-08-07T00:19:51.854777Z",
-     "shell.execute_reply": "2023-08-07T00:19:51.854196Z"
+     "iopub.execute_input": "2023-08-21T02:29:19.007847Z",
+     "iopub.status.busy": "2023-08-21T02:29:19.007564Z",
+     "iopub.status.idle": "2023-08-21T02:29:19.010742Z",
+     "shell.execute_reply": "2023-08-21T02:29:19.010371Z"
     }
    },
    "outputs": [
@@ -916,7 +918,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4f73096d",
+   "id": "b4662444",
    "metadata": {},
    "source": [
     "The  *q-values* output by the\n",
@@ -932,13 +934,13 @@
   {
    "cell_type": "code",
    "execution_count": 19,
-   "id": "bfa39f7c",
+   "id": "b00da3a1",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:51.856795Z",
-     "iopub.status.busy": "2023-08-07T00:19:51.856678Z",
-     "iopub.status.idle": "2023-08-07T00:19:51.859719Z",
-     "shell.execute_reply": "2023-08-07T00:19:51.859327Z"
+     "iopub.execute_input": "2023-08-21T02:29:19.012400Z",
+     "iopub.status.busy": "2023-08-21T02:29:19.012298Z",
+     "iopub.status.idle": "2023-08-21T02:29:19.015314Z",
+     "shell.execute_reply": "2023-08-21T02:29:19.014978Z"
     },
     "lines_to_next_cell": 0
    },
@@ -960,7 +962,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ccb44c8d",
+   "id": "fdccb808",
    "metadata": {},
    "source": [
     "We find that 146 of the 2,000 fund managers have a $q$-value below\n",
@@ -976,13 +978,13 @@
   {
    "cell_type": "code",
    "execution_count": 20,
-   "id": "70b69b47",
+   "id": "1c230117",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:51.861924Z",
-     "iopub.status.busy": "2023-08-07T00:19:51.861522Z",
-     "iopub.status.idle": "2023-08-07T00:19:51.864394Z",
-     "shell.execute_reply": "2023-08-07T00:19:51.863987Z"
+     "iopub.execute_input": "2023-08-21T02:29:19.016857Z",
+     "iopub.status.busy": "2023-08-21T02:29:19.016769Z",
+     "iopub.status.idle": "2023-08-21T02:29:19.019332Z",
+     "shell.execute_reply": "2023-08-21T02:29:19.019032Z"
     },
     "lines_to_next_cell": 2
    },
@@ -1004,7 +1006,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c8a969f4",
+   "id": "6112239d",
    "metadata": {},
    "source": [
     "Figure 13.6 displays the ordered\n",
@@ -1026,13 +1028,13 @@
   {
    "cell_type": "code",
    "execution_count": 21,
-   "id": "4c0ddea1",
+   "id": "62289650",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:51.865978Z",
-     "iopub.status.busy": "2023-08-07T00:19:51.865869Z",
-     "iopub.status.idle": "2023-08-07T00:19:51.868792Z",
-     "shell.execute_reply": "2023-08-07T00:19:51.868357Z"
+     "iopub.execute_input": "2023-08-21T02:29:19.021112Z",
+     "iopub.status.busy": "2023-08-21T02:29:19.020904Z",
+     "iopub.status.idle": "2023-08-21T02:29:19.023622Z",
+     "shell.execute_reply": "2023-08-21T02:29:19.023338Z"
     }
    },
    "outputs": [],
@@ -1051,7 +1053,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ddeb3900",
+   "id": "c36b13b7",
    "metadata": {},
    "source": [
     "We now reproduce  the middle panel of Figure 13.6."
@@ -1060,13 +1062,13 @@
   {
    "cell_type": "code",
    "execution_count": 22,
-   "id": "0314eac9",
+   "id": "18b3c0ed",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:51.871473Z",
-     "iopub.status.busy": "2023-08-07T00:19:51.871214Z",
-     "iopub.status.idle": "2023-08-07T00:19:52.126671Z",
-     "shell.execute_reply": "2023-08-07T00:19:52.126261Z"
+     "iopub.execute_input": "2023-08-21T02:29:19.025191Z",
+     "iopub.status.busy": "2023-08-21T02:29:19.025074Z",
+     "iopub.status.idle": "2023-08-21T02:29:19.262207Z",
+     "shell.execute_reply": "2023-08-21T02:29:19.261823Z"
     },
     "lines_to_next_cell": 2
    },
@@ -1096,7 +1098,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "83416f4a",
+   "id": "d87198e4",
    "metadata": {},
    "source": [
     "## A Re-Sampling Approach\n",
@@ -1110,13 +1112,13 @@
   {
    "cell_type": "code",
    "execution_count": 23,
-   "id": "b59b8137",
+   "id": "eb79e606",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:52.129167Z",
-     "iopub.status.busy": "2023-08-07T00:19:52.128842Z",
-     "iopub.status.idle": "2023-08-07T00:19:52.208320Z",
-     "shell.execute_reply": "2023-08-07T00:19:52.207936Z"
+     "iopub.execute_input": "2023-08-21T02:29:19.264174Z",
+     "iopub.status.busy": "2023-08-21T02:29:19.264030Z",
+     "iopub.status.idle": "2023-08-21T02:29:19.339232Z",
+     "shell.execute_reply": "2023-08-21T02:29:19.338912Z"
     },
     "lines_to_next_cell": 2
    },
@@ -1145,7 +1147,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5534c8d4",
+   "id": "659ee2b8",
    "metadata": {},
    "source": [
     "There are four classes of cancer. For each gene, we compare the mean\n",
@@ -1161,13 +1163,13 @@
   {
    "cell_type": "code",
    "execution_count": 24,
-   "id": "96fb2f61",
+   "id": "1afbcf47",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:52.210235Z",
-     "iopub.status.busy": "2023-08-07T00:19:52.210101Z",
-     "iopub.status.idle": "2023-08-07T00:19:52.215004Z",
-     "shell.execute_reply": "2023-08-07T00:19:52.214604Z"
+     "iopub.execute_input": "2023-08-21T02:29:19.341009Z",
+     "iopub.status.busy": "2023-08-21T02:29:19.340889Z",
+     "iopub.status.idle": "2023-08-21T02:29:19.344670Z",
+     "shell.execute_reply": "2023-08-21T02:29:19.344391Z"
     },
     "lines_to_next_cell": 2
    },
@@ -1195,7 +1197,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3131124e",
+   "id": "61f24919",
    "metadata": {},
    "source": [
     "However, this $p$-value relies on the assumption that under the null\n",
@@ -1214,13 +1216,13 @@
   {
    "cell_type": "code",
    "execution_count": 25,
-   "id": "fdc229fa",
+   "id": "f73f4c6d",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:52.217206Z",
-     "iopub.status.busy": "2023-08-07T00:19:52.217085Z",
-     "iopub.status.idle": "2023-08-07T00:19:54.776066Z",
-     "shell.execute_reply": "2023-08-07T00:19:54.775642Z"
+     "iopub.execute_input": "2023-08-21T02:29:19.346368Z",
+     "iopub.status.busy": "2023-08-21T02:29:19.346227Z",
+     "iopub.status.idle": "2023-08-21T02:29:21.776569Z",
+     "shell.execute_reply": "2023-08-21T02:29:21.776267Z"
     },
     "lines_to_next_cell": 2
    },
@@ -1253,7 +1255,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c7fc4557",
+   "id": "a97f74af",
    "metadata": {},
    "source": [
     "This fraction, 0.0398,\n",
@@ -1265,13 +1267,13 @@
   {
    "cell_type": "code",
    "execution_count": 26,
-   "id": "e3894695",
+   "id": "062daf19",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:54.778563Z",
-     "iopub.status.busy": "2023-08-07T00:19:54.778388Z",
-     "iopub.status.idle": "2023-08-07T00:19:55.017161Z",
-     "shell.execute_reply": "2023-08-07T00:19:55.016821Z"
+     "iopub.execute_input": "2023-08-21T02:29:21.778366Z",
+     "iopub.status.busy": "2023-08-21T02:29:21.778242Z",
+     "iopub.status.idle": "2023-08-21T02:29:21.990476Z",
+     "shell.execute_reply": "2023-08-21T02:29:21.989965Z"
     },
     "lines_to_next_cell": 0
    },
@@ -1307,7 +1309,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3bd21158",
+   "id": "e81b939b",
    "metadata": {},
    "source": [
     "The re-sampling-based null distribution is almost identical to the theoretical null distribution, which is displayed in red.\n",
@@ -1325,13 +1327,13 @@
   {
    "cell_type": "code",
    "execution_count": 27,
-   "id": "3b7392cb",
+   "id": "6d14fcad",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:19:55.019036Z",
-     "iopub.status.busy": "2023-08-07T00:19:55.018920Z",
-     "iopub.status.idle": "2023-08-07T00:39:19.291005Z",
-     "shell.execute_reply": "2023-08-07T00:39:19.287314Z"
+     "iopub.execute_input": "2023-08-21T02:29:21.992665Z",
+     "iopub.status.busy": "2023-08-21T02:29:21.992515Z",
+     "iopub.status.idle": "2023-08-21T02:34:05.930300Z",
+     "shell.execute_reply": "2023-08-21T02:34:05.929181Z"
     }
    },
    "outputs": [],
@@ -1358,7 +1360,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1b92df1b",
+   "id": "06286699",
    "metadata": {},
    "source": [
     "Next, we compute the number of rejected null hypotheses $R$, the\n",
@@ -1371,13 +1373,13 @@
   {
    "cell_type": "code",
    "execution_count": 28,
-   "id": "cac15616",
+   "id": "8f0ec909",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:39:19.314420Z",
-     "iopub.status.busy": "2023-08-07T00:39:19.313452Z",
-     "iopub.status.idle": "2023-08-07T00:39:19.544251Z",
-     "shell.execute_reply": "2023-08-07T00:39:19.543932Z"
+     "iopub.execute_input": "2023-08-21T02:34:05.935513Z",
+     "iopub.status.busy": "2023-08-21T02:34:05.935323Z",
+     "iopub.status.idle": "2023-08-21T02:34:06.118079Z",
+     "shell.execute_reply": "2023-08-21T02:34:06.117633Z"
     }
    },
    "outputs": [],
@@ -1394,7 +1396,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f6779ea0",
+   "id": "e26b64c6",
    "metadata": {},
    "source": [
     "Now, for any given FDR, we can find the genes that will be\n",
@@ -1412,13 +1414,13 @@
   {
    "cell_type": "code",
    "execution_count": 29,
-   "id": "9661eb10",
+   "id": "f11339e5",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:39:19.546693Z",
-     "iopub.status.busy": "2023-08-07T00:39:19.546543Z",
-     "iopub.status.idle": "2023-08-07T00:39:19.549970Z",
-     "shell.execute_reply": "2023-08-07T00:39:19.549697Z"
+     "iopub.execute_input": "2023-08-21T02:34:06.120138Z",
+     "iopub.status.busy": "2023-08-21T02:34:06.119994Z",
+     "iopub.status.idle": "2023-08-21T02:34:06.123846Z",
+     "shell.execute_reply": "2023-08-21T02:34:06.123478Z"
     }
    },
    "outputs": [
@@ -1457,7 +1459,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "001e3fc1",
+   "id": "e145621b",
    "metadata": {},
    "source": [
     "At an FDR threshold of 0.2, more genes are selected, at the cost of having a higher expected\n",
@@ -1467,13 +1469,13 @@
   {
    "cell_type": "code",
    "execution_count": 30,
-   "id": "18ad4900",
+   "id": "d2600773",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:39:19.552090Z",
-     "iopub.status.busy": "2023-08-07T00:39:19.552004Z",
-     "iopub.status.idle": "2023-08-07T00:39:19.554743Z",
-     "shell.execute_reply": "2023-08-07T00:39:19.554473Z"
+     "iopub.execute_input": "2023-08-21T02:34:06.126460Z",
+     "iopub.status.busy": "2023-08-21T02:34:06.126346Z",
+     "iopub.status.idle": "2023-08-21T02:34:06.129561Z",
+     "shell.execute_reply": "2023-08-21T02:34:06.129124Z"
     }
    },
    "outputs": [
@@ -1524,7 +1526,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8767f70c",
+   "id": "32e600ff",
    "metadata": {},
    "source": [
     "The next line  generates  Figure 13.11, which is similar\n",
@@ -1535,13 +1537,13 @@
   {
    "cell_type": "code",
    "execution_count": 31,
-   "id": "28c276b6",
+   "id": "924b7705",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2023-08-07T00:39:19.556715Z",
-     "iopub.status.busy": "2023-08-07T00:39:19.556515Z",
-     "iopub.status.idle": "2023-08-07T00:39:19.650514Z",
-     "shell.execute_reply": "2023-08-07T00:39:19.650181Z"
+     "iopub.execute_input": "2023-08-21T02:34:06.131323Z",
+     "iopub.status.busy": "2023-08-21T02:34:06.131207Z",
+     "iopub.status.idle": "2023-08-21T02:34:06.216626Z",
+     "shell.execute_reply": "2023-08-21T02:34:06.216270Z"
     },
     "lines_to_next_cell": 0
    },
@@ -1566,18 +1568,18 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e4b5d621",
+   "id": "b9f54695",
    "metadata": {},
    "source": [
-    "\n"
+    " \n"
    ]
   }
  ],
  "metadata": {
   "jupytext": {
    "cell_metadata_filter": "-all",
-   "formats": "ipynb,Rmd",
-   "main_language": "python"
+   "main_language": "python",
+   "notebook_metadata_filter": "-all"
   },
   "language_info": {
    "codemirror_mode": {