Revised with 1.6.1
This commit is contained in:
daviddoji committed
1 parent
a3b5cd7deb
commit
85271e0ea6
3 files changed
+3169
No files matched your search
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,922 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Collecting experiments data in a data frame\n",
|
||||
"\n",
|
||||
"### Bogumił Kamiński"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"using DataFrames\n",
|
||||
"using Statistics\n",
|
||||
"using PyPlot\n",
|
||||
"using Random\n",
|
||||
"using Pipe"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In this part we will run a simple Monte Carlo simulation so show examples how one can work with data frames."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Consider the following puzzle**:\n",
|
||||
"\n",
|
||||
"We draw independent random numbers from $U(0,1)$ distribution. On the average, how many draws do we need, till the sum of these numbers exceeds $1$?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Here is the code that runs this experiment once. For tutorial reasons we keep all the generated random numbers and recalculate their sum in each iteration (you can try to improve the efficiency of this code as an exercise)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"sim_e (generic function with 1 method)"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"function sim_e()\n",
|
||||
" draw = Float64[]\n",
|
||||
" while true\n",
|
||||
" push!(draw, rand())\n",
|
||||
" sum(draw) > 1.0 && return draw\n",
|
||||
" end\n",
|
||||
"end"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"Random.seed!(1234); # just to make sure we get the same results if we are on the same version of Julia"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let us run our simulation several times:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"5-element Vector{Vector{Float64}}:\n",
|
||||
" [0.5908446386657102, 0.7667970365022592]\n",
|
||||
" [0.5662374165061859, 0.4600853424625171]\n",
|
||||
" [0.7940257103317943, 0.8541465903790502]\n",
|
||||
" [0.20058603493384108, 0.2986142783434118, 0.24683718661000897, 0.5796722333690416]\n",
|
||||
" [0.6488819502093455, 0.010905889635595356, 0.06642303695533736, 0.9567533636029237]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"res = [sim_e() for _ in 1:5]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"and check that each time we finished just when we exceeded $1$:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"5-element Vector{Float64}:\n",
|
||||
" 1.3576416751679694\n",
|
||||
" 1.026322758968703\n",
|
||||
" 1.6481723007108444\n",
|
||||
" 1.3257097332563035\n",
|
||||
" 1.682964240403202"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"sum.(res)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"5-element Vector{Float64}:\n",
|
||||
" 0.5908446386657102\n",
|
||||
" 0.5662374165061859\n",
|
||||
" 0.7940257103317943\n",
|
||||
" 0.7460374998872619\n",
|
||||
" 0.7262108768002782"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"@. sum(res) - last(res)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"All looks good so far! (and as a bonus we have just made a small exercise in broadcasting)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now let us populate a data frame with the results of our experiments"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 25.942720 seconds (151.90 M allocations: 4.928 GiB, 20.83% gc time, 5.57% compilation time)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"df = DataFrame()\n",
|
||||
"\n",
|
||||
"@time for i in 1:10^7\n",
|
||||
" push!(df, (id=i, pos=sim_e()))\n",
|
||||
"end"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"As you can see the process was quite fast, `push!`-ing data to a `DataFrame` is efficient."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table class=\"data-frame\"><thead><tr><th></th><th>id</th><th>pos</th></tr><tr><th></th><th>Int64</th><th>Array…</th></tr></thead><tbody><p>10,000,000 rows × 2 columns</p><tr><th>1</th><td>1</td><td>[0.646691, 0.112486, 0.276021]</td></tr><tr><th>2</th><td>2</td><td>[0.651664, 0.0566425, 0.842714]</td></tr><tr><th>3</th><td>3</td><td>[0.950498, 0.96467]</td></tr><tr><th>4</th><td>4</td><td>[0.945775, 0.789904]</td></tr><tr><th>5</th><td>5</td><td>[0.82116, 0.0341601, 0.0945445, 0.314926]</td></tr><tr><th>6</th><td>6</td><td>[0.12781, 0.374187, 0.931115]</td></tr><tr><th>7</th><td>7</td><td>[0.438939, 0.246862, 0.0118196, 0.0460428, 0.496169]</td></tr><tr><th>8</th><td>8</td><td>[0.732, 0.299058]</td></tr><tr><th>9</th><td>9</td><td>[0.449182, 0.875096]</td></tr><tr><th>10</th><td>10</td><td>[0.0462887, 0.698356, 0.365109]</td></tr><tr><th>11</th><td>11</td><td>[0.302478, 0.372575, 0.150508, 0.147329, 0.283401]</td></tr><tr><th>12</th><td>12</td><td>[0.404953, 0.499531, 0.658815]</td></tr><tr><th>13</th><td>13</td><td>[0.515627, 0.260715, 0.59552]</td></tr><tr><th>14</th><td>14</td><td>[0.292462, 0.28858, 0.61816]</td></tr><tr><th>15</th><td>15</td><td>[0.66426, 0.753508]</td></tr><tr><th>16</th><td>16</td><td>[0.0368842, 0.643704, 0.401421]</td></tr><tr><th>17</th><td>17</td><td>[0.525057, 0.61201]</td></tr><tr><th>18</th><td>18</td><td>[0.432577, 0.082207, 0.199058, 0.576082]</td></tr><tr><th>19</th><td>19</td><td>[0.218177, 0.362036, 0.204728, 0.932984]</td></tr><tr><th>20</th><td>20</td><td>[0.827263, 0.0992992, 0.6343]</td></tr><tr><th>21</th><td>21</td><td>[0.132715, 0.775194, 0.869237]</td></tr><tr><th>22</th><td>22</td><td>[0.0396356, 0.79041, 0.431188]</td></tr><tr><th>23</th><td>23</td><td>[0.137658, 0.60808, 0.255054]</td></tr><tr><th>24</th><td>24</td><td>[0.498734, 0.0940369, 0.52509]</td></tr><tr><th>25</th><td>25</td><td>[0.265511, 0.110096, 0.834362]</td></tr><tr><th>26</th><td>26</td><td>[0.633427, 0.337865, 0.112987]</td></tr><tr><th>27</th><td>27</td><td>[0.78299, 0.838042]</td></tr><tr><th>28</th><td>28</td><td>[0.0878598, 0.386568, 0.330579, 0.748041]</td></tr><tr><th>29</th><td>29</td><td>[0.265595, 0.291069, 0.612628]</td></tr><tr><th>30</th><td>30</td><td>[0.705766, 0.508363]</td></tr><tr><th>⋮</th><td>⋮</td><td>⋮</td></tr></tbody></table>"
|
||||
],
|
||||
"text/latex": [
|
||||
"\\begin{tabular}{r|cc}\n",
|
||||
"\t& id & pos\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t& Int64 & Array…\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t1 & 1 & [0.646691, 0.112486, 0.276021] \\\\\n",
|
||||
"\t2 & 2 & [0.651664, 0.0566425, 0.842714] \\\\\n",
|
||||
"\t3 & 3 & [0.950498, 0.96467] \\\\\n",
|
||||
"\t4 & 4 & [0.945775, 0.789904] \\\\\n",
|
||||
"\t5 & 5 & [0.82116, 0.0341601, 0.0945445, 0.314926] \\\\\n",
|
||||
"\t6 & 6 & [0.12781, 0.374187, 0.931115] \\\\\n",
|
||||
"\t7 & 7 & [0.438939, 0.246862, 0.0118196, 0.0460428, 0.496169] \\\\\n",
|
||||
"\t8 & 8 & [0.732, 0.299058] \\\\\n",
|
||||
"\t9 & 9 & [0.449182, 0.875096] \\\\\n",
|
||||
"\t10 & 10 & [0.0462887, 0.698356, 0.365109] \\\\\n",
|
||||
"\t11 & 11 & [0.302478, 0.372575, 0.150508, 0.147329, 0.283401] \\\\\n",
|
||||
"\t12 & 12 & [0.404953, 0.499531, 0.658815] \\\\\n",
|
||||
"\t13 & 13 & [0.515627, 0.260715, 0.59552] \\\\\n",
|
||||
"\t14 & 14 & [0.292462, 0.28858, 0.61816] \\\\\n",
|
||||
"\t15 & 15 & [0.66426, 0.753508] \\\\\n",
|
||||
"\t16 & 16 & [0.0368842, 0.643704, 0.401421] \\\\\n",
|
||||
"\t17 & 17 & [0.525057, 0.61201] \\\\\n",
|
||||
"\t18 & 18 & [0.432577, 0.082207, 0.199058, 0.576082] \\\\\n",
|
||||
"\t19 & 19 & [0.218177, 0.362036, 0.204728, 0.932984] \\\\\n",
|
||||
"\t20 & 20 & [0.827263, 0.0992992, 0.6343] \\\\\n",
|
||||
"\t21 & 21 & [0.132715, 0.775194, 0.869237] \\\\\n",
|
||||
"\t22 & 22 & [0.0396356, 0.79041, 0.431188] \\\\\n",
|
||||
"\t23 & 23 & [0.137658, 0.60808, 0.255054] \\\\\n",
|
||||
"\t24 & 24 & [0.498734, 0.0940369, 0.52509] \\\\\n",
|
||||
"\t25 & 25 & [0.265511, 0.110096, 0.834362] \\\\\n",
|
||||
"\t26 & 26 & [0.633427, 0.337865, 0.112987] \\\\\n",
|
||||
"\t27 & 27 & [0.78299, 0.838042] \\\\\n",
|
||||
"\t28 & 28 & [0.0878598, 0.386568, 0.330579, 0.748041] \\\\\n",
|
||||
"\t29 & 29 & [0.265595, 0.291069, 0.612628] \\\\\n",
|
||||
"\t30 & 30 & [0.705766, 0.508363] \\\\\n",
|
||||
"\t$\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
|
||||
"\\end{tabular}\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"\u001b[1m10000000×2 DataFrame\u001b[0m\n",
|
||||
"\u001b[1m Row \u001b[0m│\u001b[1m id \u001b[0m\u001b[1m pos \u001b[0m\n",
|
||||
"\u001b[1m \u001b[0m│\u001b[90m Int64 \u001b[0m\u001b[90m Array… \u001b[0m\n",
|
||||
"──────────┼─────────────────────────────────────────────\n",
|
||||
" 1 │ 1 [0.646691, 0.112486, 0.276021]\n",
|
||||
" 2 │ 2 [0.651664, 0.0566425, 0.842714]\n",
|
||||
" 3 │ 3 [0.950498, 0.96467]\n",
|
||||
" 4 │ 4 [0.945775, 0.789904]\n",
|
||||
" 5 │ 5 [0.82116, 0.0341601, 0.0945445, …\n",
|
||||
" 6 │ 6 [0.12781, 0.374187, 0.931115]\n",
|
||||
" 7 │ 7 [0.438939, 0.246862, 0.0118196, …\n",
|
||||
" 8 │ 8 [0.732, 0.299058]\n",
|
||||
" 9 │ 9 [0.449182, 0.875096]\n",
|
||||
" 10 │ 10 [0.0462887, 0.698356, 0.365109]\n",
|
||||
" 11 │ 11 [0.302478, 0.372575, 0.150508, 0…\n",
|
||||
" ⋮ │ ⋮ ⋮\n",
|
||||
" 9999991 │ 9999991 [0.700468, 0.220524, 0.347931]\n",
|
||||
" 9999992 │ 9999992 [0.231368, 0.862016]\n",
|
||||
" 9999993 │ 9999993 [0.869351, 0.444795]\n",
|
||||
" 9999994 │ 9999994 [0.821356, 0.509054]\n",
|
||||
" 9999995 │ 9999995 [0.589245, 0.669708]\n",
|
||||
" 9999996 │ 9999996 [0.806262, 0.734397]\n",
|
||||
" 9999997 │ 9999997 [0.216506, 0.430571, 0.283787, 0…\n",
|
||||
" 9999998 │ 9999998 [0.0100723, 0.836315, 0.942299]\n",
|
||||
" 9999999 │ 9999999 [0.499669, 0.25214, 0.964065]\n",
|
||||
" 10000000 │ 10000000 [0.663339, 0.887989]\n",
|
||||
"\u001b[36m 9999979 rows omitted\u001b[0m"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"df"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let us count the number of jumps we have made in each step using the `transform!` function:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table class=\"data-frame\"><thead><tr><th></th><th>id</th><th>pos</th><th>jumps</th></tr><tr><th></th><th>Int64</th><th>Array…</th><th>Int64</th></tr></thead><tbody><p>10,000,000 rows × 3 columns</p><tr><th>1</th><td>1</td><td>[0.646691, 0.112486, 0.276021]</td><td>3</td></tr><tr><th>2</th><td>2</td><td>[0.651664, 0.0566425, 0.842714]</td><td>3</td></tr><tr><th>3</th><td>3</td><td>[0.950498, 0.96467]</td><td>2</td></tr><tr><th>4</th><td>4</td><td>[0.945775, 0.789904]</td><td>2</td></tr><tr><th>5</th><td>5</td><td>[0.82116, 0.0341601, 0.0945445, 0.314926]</td><td>4</td></tr><tr><th>6</th><td>6</td><td>[0.12781, 0.374187, 0.931115]</td><td>3</td></tr><tr><th>7</th><td>7</td><td>[0.438939, 0.246862, 0.0118196, 0.0460428, 0.496169]</td><td>5</td></tr><tr><th>8</th><td>8</td><td>[0.732, 0.299058]</td><td>2</td></tr><tr><th>9</th><td>9</td><td>[0.449182, 0.875096]</td><td>2</td></tr><tr><th>10</th><td>10</td><td>[0.0462887, 0.698356, 0.365109]</td><td>3</td></tr><tr><th>11</th><td>11</td><td>[0.302478, 0.372575, 0.150508, 0.147329, 0.283401]</td><td>5</td></tr><tr><th>12</th><td>12</td><td>[0.404953, 0.499531, 0.658815]</td><td>3</td></tr><tr><th>13</th><td>13</td><td>[0.515627, 0.260715, 0.59552]</td><td>3</td></tr><tr><th>14</th><td>14</td><td>[0.292462, 0.28858, 0.61816]</td><td>3</td></tr><tr><th>15</th><td>15</td><td>[0.66426, 0.753508]</td><td>2</td></tr><tr><th>16</th><td>16</td><td>[0.0368842, 0.643704, 0.401421]</td><td>3</td></tr><tr><th>17</th><td>17</td><td>[0.525057, 0.61201]</td><td>2</td></tr><tr><th>18</th><td>18</td><td>[0.432577, 0.082207, 0.199058, 0.576082]</td><td>4</td></tr><tr><th>19</th><td>19</td><td>[0.218177, 0.362036, 0.204728, 0.932984]</td><td>4</td></tr><tr><th>20</th><td>20</td><td>[0.827263, 0.0992992, 0.6343]</td><td>3</td></tr><tr><th>21</th><td>21</td><td>[0.132715, 0.775194, 0.869237]</td><td>3</td></tr><tr><th>22</th><td>22</td><td>[0.0396356, 0.79041, 0.431188]</td><td>3</td></tr><tr><th>23</th><td>23</td><td>[0.137658, 0.60808, 0.255054]</td><td>3</td></tr><tr><th>24</th><td>24</td><td>[0.498734, 0.0940369, 0.52509]</td><td>3</td></tr><tr><th>25</th><td>25</td><td>[0.265511, 0.110096, 0.834362]</td><td>3</td></tr><tr><th>26</th><td>26</td><td>[0.633427, 0.337865, 0.112987]</td><td>3</td></tr><tr><th>27</th><td>27</td><td>[0.78299, 0.838042]</td><td>2</td></tr><tr><th>28</th><td>28</td><td>[0.0878598, 0.386568, 0.330579, 0.748041]</td><td>4</td></tr><tr><th>29</th><td>29</td><td>[0.265595, 0.291069, 0.612628]</td><td>3</td></tr><tr><th>30</th><td>30</td><td>[0.705766, 0.508363]</td><td>2</td></tr><tr><th>⋮</th><td>⋮</td><td>⋮</td><td>⋮</td></tr></tbody></table>"
|
||||
],
|
||||
"text/latex": [
|
||||
"\\begin{tabular}{r|ccc}\n",
|
||||
"\t& id & pos & jumps\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t& Int64 & Array… & Int64\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t1 & 1 & [0.646691, 0.112486, 0.276021] & 3 \\\\\n",
|
||||
"\t2 & 2 & [0.651664, 0.0566425, 0.842714] & 3 \\\\\n",
|
||||
"\t3 & 3 & [0.950498, 0.96467] & 2 \\\\\n",
|
||||
"\t4 & 4 & [0.945775, 0.789904] & 2 \\\\\n",
|
||||
"\t5 & 5 & [0.82116, 0.0341601, 0.0945445, 0.314926] & 4 \\\\\n",
|
||||
"\t6 & 6 & [0.12781, 0.374187, 0.931115] & 3 \\\\\n",
|
||||
"\t7 & 7 & [0.438939, 0.246862, 0.0118196, 0.0460428, 0.496169] & 5 \\\\\n",
|
||||
"\t8 & 8 & [0.732, 0.299058] & 2 \\\\\n",
|
||||
"\t9 & 9 & [0.449182, 0.875096] & 2 \\\\\n",
|
||||
"\t10 & 10 & [0.0462887, 0.698356, 0.365109] & 3 \\\\\n",
|
||||
"\t11 & 11 & [0.302478, 0.372575, 0.150508, 0.147329, 0.283401] & 5 \\\\\n",
|
||||
"\t12 & 12 & [0.404953, 0.499531, 0.658815] & 3 \\\\\n",
|
||||
"\t13 & 13 & [0.515627, 0.260715, 0.59552] & 3 \\\\\n",
|
||||
"\t14 & 14 & [0.292462, 0.28858, 0.61816] & 3 \\\\\n",
|
||||
"\t15 & 15 & [0.66426, 0.753508] & 2 \\\\\n",
|
||||
"\t16 & 16 & [0.0368842, 0.643704, 0.401421] & 3 \\\\\n",
|
||||
"\t17 & 17 & [0.525057, 0.61201] & 2 \\\\\n",
|
||||
"\t18 & 18 & [0.432577, 0.082207, 0.199058, 0.576082] & 4 \\\\\n",
|
||||
"\t19 & 19 & [0.218177, 0.362036, 0.204728, 0.932984] & 4 \\\\\n",
|
||||
"\t20 & 20 & [0.827263, 0.0992992, 0.6343] & 3 \\\\\n",
|
||||
"\t21 & 21 & [0.132715, 0.775194, 0.869237] & 3 \\\\\n",
|
||||
"\t22 & 22 & [0.0396356, 0.79041, 0.431188] & 3 \\\\\n",
|
||||
"\t23 & 23 & [0.137658, 0.60808, 0.255054] & 3 \\\\\n",
|
||||
"\t24 & 24 & [0.498734, 0.0940369, 0.52509] & 3 \\\\\n",
|
||||
"\t25 & 25 & [0.265511, 0.110096, 0.834362] & 3 \\\\\n",
|
||||
"\t26 & 26 & [0.633427, 0.337865, 0.112987] & 3 \\\\\n",
|
||||
"\t27 & 27 & [0.78299, 0.838042] & 2 \\\\\n",
|
||||
"\t28 & 28 & [0.0878598, 0.386568, 0.330579, 0.748041] & 4 \\\\\n",
|
||||
"\t29 & 29 & [0.265595, 0.291069, 0.612628] & 3 \\\\\n",
|
||||
"\t30 & 30 & [0.705766, 0.508363] & 2 \\\\\n",
|
||||
"\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
|
||||
"\\end{tabular}\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"\u001b[1m10000000×3 DataFrame\u001b[0m\n",
|
||||
"\u001b[1m Row \u001b[0m│\u001b[1m id \u001b[0m\u001b[1m pos \u001b[0m\u001b[1m jumps \u001b[0m\n",
|
||||
"\u001b[1m \u001b[0m│\u001b[90m Int64 \u001b[0m\u001b[90m Array… \u001b[0m\u001b[90m Int64 \u001b[0m\n",
|
||||
"──────────┼────────────────────────────────────────────────────\n",
|
||||
" 1 │ 1 [0.646691, 0.112486, 0.276021] 3\n",
|
||||
" 2 │ 2 [0.651664, 0.0566425, 0.842714] 3\n",
|
||||
" 3 │ 3 [0.950498, 0.96467] 2\n",
|
||||
" 4 │ 4 [0.945775, 0.789904] 2\n",
|
||||
" 5 │ 5 [0.82116, 0.0341601, 0.0945445, … 4\n",
|
||||
" 6 │ 6 [0.12781, 0.374187, 0.931115] 3\n",
|
||||
" 7 │ 7 [0.438939, 0.246862, 0.0118196, … 5\n",
|
||||
" 8 │ 8 [0.732, 0.299058] 2\n",
|
||||
" 9 │ 9 [0.449182, 0.875096] 2\n",
|
||||
" 10 │ 10 [0.0462887, 0.698356, 0.365109] 3\n",
|
||||
" 11 │ 11 [0.302478, 0.372575, 0.150508, 0… 5\n",
|
||||
" ⋮ │ ⋮ ⋮ ⋮\n",
|
||||
" 9999991 │ 9999991 [0.700468, 0.220524, 0.347931] 3\n",
|
||||
" 9999992 │ 9999992 [0.231368, 0.862016] 2\n",
|
||||
" 9999993 │ 9999993 [0.869351, 0.444795] 2\n",
|
||||
" 9999994 │ 9999994 [0.821356, 0.509054] 2\n",
|
||||
" 9999995 │ 9999995 [0.589245, 0.669708] 2\n",
|
||||
" 9999996 │ 9999996 [0.806262, 0.734397] 2\n",
|
||||
" 9999997 │ 9999997 [0.216506, 0.430571, 0.283787, 0… 4\n",
|
||||
" 9999998 │ 9999998 [0.0100723, 0.836315, 0.942299] 3\n",
|
||||
" 9999999 │ 9999999 [0.499669, 0.25214, 0.964065] 3\n",
|
||||
" 10000000 │ 10000000 [0.663339, 0.887989] 2\n",
|
||||
"\u001b[36m 9999979 rows omitted\u001b[0m"
|
||||
]
|
||||
},
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"transform!(df, :pos => ByRow(length) => :jumps)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let us dissect what we have written above:\n",
|
||||
"* `transform!` adds columns to a data frame in-place\n",
|
||||
"* `:pos` is a source column\n",
|
||||
"* `ByRow(length)` tells us that we want to apply `length` function to each element for `:pos` column (without it `length` would be applied to the whole column - can you guess what would be the result?)\n",
|
||||
"* `:jumps` is the name of the column that should be created"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now we are ready to find the average number of jumps that are made:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"2.7185991"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"mean(df.jumps)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"or"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table class=\"data-frame\"><thead><tr><th></th><th>jumps_mean</th></tr><tr><th></th><th>Float64</th></tr></thead><tbody><p>1 rows × 1 columns</p><tr><th>1</th><td>2.7186</td></tr></tbody></table>"
|
||||
],
|
||||
"text/latex": [
|
||||
"\\begin{tabular}{r|c}\n",
|
||||
"\t& jumps\\_mean\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t& Float64\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t1 & 2.7186 \\\\\n",
|
||||
"\\end{tabular}\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"\u001b[1m1×1 DataFrame\u001b[0m\n",
|
||||
"\u001b[1m Row \u001b[0m│\u001b[1m jumps_mean \u001b[0m\n",
|
||||
"\u001b[1m \u001b[0m│\u001b[90m Float64 \u001b[0m\n",
|
||||
"─────┼────────────\n",
|
||||
" 1 │ 2.7186"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"combine(df, :jumps => mean)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"which happens to be very close to:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"ℯ = 2.7182818284590..."
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"MathConstants.e"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let us now find a distribution of number of jumps:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table class=\"data-frame\"><thead><tr><th></th><th>jumps</th><th>jumps_length</th></tr><tr><th></th><th>Int64</th><th>Int64</th></tr></thead><tbody><p>10 rows × 2 columns</p><tr><th>1</th><td>2</td><td>4999743</td></tr><tr><th>2</th><td>3</td><td>3332539</td></tr><tr><th>3</th><td>4</td><td>1250009</td></tr><tr><th>4</th><td>5</td><td>333738</td></tr><tr><th>5</th><td>6</td><td>69865</td></tr><tr><th>6</th><td>7</td><td>12145</td></tr><tr><th>7</th><td>8</td><td>1725</td></tr><tr><th>8</th><td>9</td><td>204</td></tr><tr><th>9</th><td>10</td><td>31</td></tr><tr><th>10</th><td>11</td><td>1</td></tr></tbody></table>"
|
||||
],
|
||||
"text/latex": [
|
||||
"\\begin{tabular}{r|cc}\n",
|
||||
"\t& jumps & jumps\\_length\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t& Int64 & Int64\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t1 & 2 & 4999743 \\\\\n",
|
||||
"\t2 & 3 & 3332539 \\\\\n",
|
||||
"\t3 & 4 & 1250009 \\\\\n",
|
||||
"\t4 & 5 & 333738 \\\\\n",
|
||||
"\t5 & 6 & 69865 \\\\\n",
|
||||
"\t6 & 7 & 12145 \\\\\n",
|
||||
"\t7 & 8 & 1725 \\\\\n",
|
||||
"\t8 & 9 & 204 \\\\\n",
|
||||
"\t9 & 10 & 31 \\\\\n",
|
||||
"\t10 & 11 & 1 \\\\\n",
|
||||
"\\end{tabular}\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"\u001b[1m10×2 DataFrame\u001b[0m\n",
|
||||
"\u001b[1m Row \u001b[0m│\u001b[1m jumps \u001b[0m\u001b[1m jumps_length \u001b[0m\n",
|
||||
"\u001b[1m \u001b[0m│\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\n",
|
||||
"─────┼─────────────────────\n",
|
||||
" 1 │ 2 4999743\n",
|
||||
" 2 │ 3 3332539\n",
|
||||
" 3 │ 4 1250009\n",
|
||||
" 4 │ 5 333738\n",
|
||||
" 5 │ 6 69865\n",
|
||||
" 6 │ 7 12145\n",
|
||||
" 7 │ 8 1725\n",
|
||||
" 8 │ 9 204\n",
|
||||
" 9 │ 10 31\n",
|
||||
" 10 │ 11 1"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"jumps_agg = @pipe df |>\n",
|
||||
" groupby(_, :jumps, sort=true) |>\n",
|
||||
" combine(_, :jumps => length)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"and normalize it as a fraction (and at the same time calculate some theoretical result that we have *guessed* :)):"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table class=\"data-frame\"><thead><tr><th></th><th>jumps</th><th>jumps_length</th><th>simulation</th><th>theory</th></tr><tr><th></th><th>Int64</th><th>Int64</th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>10 rows × 4 columns</p><tr><th>1</th><td>2</td><td>4999743</td><td>0.499974</td><td>0.5</td></tr><tr><th>2</th><td>3</td><td>3332539</td><td>0.333254</td><td>0.333333</td></tr><tr><th>3</th><td>4</td><td>1250009</td><td>0.125001</td><td>0.125</td></tr><tr><th>4</th><td>5</td><td>333738</td><td>0.0333738</td><td>0.0333333</td></tr><tr><th>5</th><td>6</td><td>69865</td><td>0.0069865</td><td>0.00694444</td></tr><tr><th>6</th><td>7</td><td>12145</td><td>0.0012145</td><td>0.00119048</td></tr><tr><th>7</th><td>8</td><td>1725</td><td>0.0001725</td><td>0.000173611</td></tr><tr><th>8</th><td>9</td><td>204</td><td>2.04e-5</td><td>2.20459e-5</td></tr><tr><th>9</th><td>10</td><td>31</td><td>3.1e-6</td><td>2.48016e-6</td></tr><tr><th>10</th><td>11</td><td>1</td><td>1.0e-7</td><td>2.50521e-7</td></tr></tbody></table>"
|
||||
],
|
||||
"text/latex": [
|
||||
"\\begin{tabular}{r|cccc}\n",
|
||||
"\t& jumps & jumps\\_length & simulation & theory\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t& Int64 & Int64 & Float64 & Float64\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t1 & 2 & 4999743 & 0.499974 & 0.5 \\\\\n",
|
||||
"\t2 & 3 & 3332539 & 0.333254 & 0.333333 \\\\\n",
|
||||
"\t3 & 4 & 1250009 & 0.125001 & 0.125 \\\\\n",
|
||||
"\t4 & 5 & 333738 & 0.0333738 & 0.0333333 \\\\\n",
|
||||
"\t5 & 6 & 69865 & 0.0069865 & 0.00694444 \\\\\n",
|
||||
"\t6 & 7 & 12145 & 0.0012145 & 0.00119048 \\\\\n",
|
||||
"\t7 & 8 & 1725 & 0.0001725 & 0.000173611 \\\\\n",
|
||||
"\t8 & 9 & 204 & 2.04e-5 & 2.20459e-5 \\\\\n",
|
||||
"\t9 & 10 & 31 & 3.1e-6 & 2.48016e-6 \\\\\n",
|
||||
"\t10 & 11 & 1 & 1.0e-7 & 2.50521e-7 \\\\\n",
|
||||
"\\end{tabular}\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"\u001b[1m10×4 DataFrame\u001b[0m\n",
|
||||
"\u001b[1m Row \u001b[0m│\u001b[1m jumps \u001b[0m\u001b[1m jumps_length \u001b[0m\u001b[1m simulation \u001b[0m\u001b[1m theory \u001b[0m\n",
|
||||
"\u001b[1m \u001b[0m│\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\n",
|
||||
"─────┼──────────────────────────────────────────────\n",
|
||||
" 1 │ 2 4999743 0.499974 0.5\n",
|
||||
" 2 │ 3 3332539 0.333254 0.333333\n",
|
||||
" 3 │ 4 1250009 0.125001 0.125\n",
|
||||
" 4 │ 5 333738 0.0333738 0.0333333\n",
|
||||
" 5 │ 6 69865 0.0069865 0.00694444\n",
|
||||
" 6 │ 7 12145 0.0012145 0.00119048\n",
|
||||
" 7 │ 8 1725 0.0001725 0.000173611\n",
|
||||
" 8 │ 9 204 2.04e-5 2.20459e-5\n",
|
||||
" 9 │ 10 31 3.1e-6 2.48016e-6\n",
|
||||
" 10 │ 11 1 1.0e-7 2.50521e-7"
|
||||
]
|
||||
},
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"transform!(jumps_agg,\n",
|
||||
" :jumps_length => (x -> x ./ sum(x)) => :simulation,\n",
|
||||
" :jumps => ByRow(x -> (x-1) / factorial(x)) => :theory)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let us briefly justify how we have guessed it (you can safely skip the derivation):\n",
|
||||
"\n",
|
||||
"Formula\n",
|
||||
"$$\n",
|
||||
"p_n = \\frac{n-1}{n!}\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\sum_{n=2}^{+\\infty}p_n=\\sum_{n=2}^{+\\infty} \\frac{n-1}{n!} = \\sum_{n=1}^{+\\infty} \\frac{1}{n!} - \\sum_{n=2}^{+\\infty} \\frac{1}{n!} = 1\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\sum_{n=2}^{+\\infty}n\\cdot p_n=\\sum_{n=2}^{+\\infty} n\\frac{n-1}{n!} = \\sum_{n=2}^{+\\infty} \\frac{1}{(n-2)!} = e\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"Now we note that:\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"1-\\sum_{n=2}^k p_n = \\frac{1}{k!}\n",
|
||||
"$$\n",
|
||||
"which can be most easily justified by a geometric argument."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"To finish this section of the tutorial let us check if random numbers generated using `rand()` were indeed $U(0,1)$."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"To do this we will add some columns to `df` data frame."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table class=\"data-frame\"><thead><tr><th></th><th>id</th><th>pos</th><th>jumps</th></tr><tr><th></th><th>Int64</th><th>Array…</th><th>Int64</th></tr></thead><tbody><p>10,000,000 rows × 3 columns</p><tr><th>1</th><td>1</td><td>[0.646691, 0.112486, 0.276021]</td><td>3</td></tr><tr><th>2</th><td>2</td><td>[0.651664, 0.0566425, 0.842714]</td><td>3</td></tr><tr><th>3</th><td>3</td><td>[0.950498, 0.96467]</td><td>2</td></tr><tr><th>4</th><td>4</td><td>[0.945775, 0.789904]</td><td>2</td></tr><tr><th>5</th><td>5</td><td>[0.82116, 0.0341601, 0.0945445, 0.314926]</td><td>4</td></tr><tr><th>6</th><td>6</td><td>[0.12781, 0.374187, 0.931115]</td><td>3</td></tr><tr><th>7</th><td>7</td><td>[0.438939, 0.246862, 0.0118196, 0.0460428, 0.496169]</td><td>5</td></tr><tr><th>8</th><td>8</td><td>[0.732, 0.299058]</td><td>2</td></tr><tr><th>9</th><td>9</td><td>[0.449182, 0.875096]</td><td>2</td></tr><tr><th>10</th><td>10</td><td>[0.0462887, 0.698356, 0.365109]</td><td>3</td></tr><tr><th>11</th><td>11</td><td>[0.302478, 0.372575, 0.150508, 0.147329, 0.283401]</td><td>5</td></tr><tr><th>12</th><td>12</td><td>[0.404953, 0.499531, 0.658815]</td><td>3</td></tr><tr><th>13</th><td>13</td><td>[0.515627, 0.260715, 0.59552]</td><td>3</td></tr><tr><th>14</th><td>14</td><td>[0.292462, 0.28858, 0.61816]</td><td>3</td></tr><tr><th>15</th><td>15</td><td>[0.66426, 0.753508]</td><td>2</td></tr><tr><th>16</th><td>16</td><td>[0.0368842, 0.643704, 0.401421]</td><td>3</td></tr><tr><th>17</th><td>17</td><td>[0.525057, 0.61201]</td><td>2</td></tr><tr><th>18</th><td>18</td><td>[0.432577, 0.082207, 0.199058, 0.576082]</td><td>4</td></tr><tr><th>19</th><td>19</td><td>[0.218177, 0.362036, 0.204728, 0.932984]</td><td>4</td></tr><tr><th>20</th><td>20</td><td>[0.827263, 0.0992992, 0.6343]</td><td>3</td></tr><tr><th>21</th><td>21</td><td>[0.132715, 0.775194, 0.869237]</td><td>3</td></tr><tr><th>22</th><td>22</td><td>[0.0396356, 0.79041, 0.431188]</td><td>3</td></tr><tr><th>23</th><td>23</td><td>[0.137658, 0.60808, 0.255054]</td><td>3</td></tr><tr><th>24</th><td>24</td><td>[0.498734, 0.0940369, 0.52509]</td><td>3</td></tr><tr><th>25</th><td>25</td><td>[0.265511, 0.110096, 0.834362]</td><td>3</td></tr><tr><th>26</th><td>26</td><td>[0.633427, 0.337865, 0.112987]</td><td>3</td></tr><tr><th>27</th><td>27</td><td>[0.78299, 0.838042]</td><td>2</td></tr><tr><th>28</th><td>28</td><td>[0.0878598, 0.386568, 0.330579, 0.748041]</td><td>4</td></tr><tr><th>29</th><td>29</td><td>[0.265595, 0.291069, 0.612628]</td><td>3</td></tr><tr><th>30</th><td>30</td><td>[0.705766, 0.508363]</td><td>2</td></tr><tr><th>⋮</th><td>⋮</td><td>⋮</td><td>⋮</td></tr></tbody></table>"
|
||||
],
|
||||
"text/latex": [
|
||||
"\\begin{tabular}{r|ccc}\n",
|
||||
"\t& id & pos & jumps\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t& Int64 & Array… & Int64\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t1 & 1 & [0.646691, 0.112486, 0.276021] & 3 \\\\\n",
|
||||
"\t2 & 2 & [0.651664, 0.0566425, 0.842714] & 3 \\\\\n",
|
||||
"\t3 & 3 & [0.950498, 0.96467] & 2 \\\\\n",
|
||||
"\t4 & 4 & [0.945775, 0.789904] & 2 \\\\\n",
|
||||
"\t5 & 5 & [0.82116, 0.0341601, 0.0945445, 0.314926] & 4 \\\\\n",
|
||||
"\t6 & 6 & [0.12781, 0.374187, 0.931115] & 3 \\\\\n",
|
||||
"\t7 & 7 & [0.438939, 0.246862, 0.0118196, 0.0460428, 0.496169] & 5 \\\\\n",
|
||||
"\t8 & 8 & [0.732, 0.299058] & 2 \\\\\n",
|
||||
"\t9 & 9 & [0.449182, 0.875096] & 2 \\\\\n",
|
||||
"\t10 & 10 & [0.0462887, 0.698356, 0.365109] & 3 \\\\\n",
|
||||
"\t11 & 11 & [0.302478, 0.372575, 0.150508, 0.147329, 0.283401] & 5 \\\\\n",
|
||||
"\t12 & 12 & [0.404953, 0.499531, 0.658815] & 3 \\\\\n",
|
||||
"\t13 & 13 & [0.515627, 0.260715, 0.59552] & 3 \\\\\n",
|
||||
"\t14 & 14 & [0.292462, 0.28858, 0.61816] & 3 \\\\\n",
|
||||
"\t15 & 15 & [0.66426, 0.753508] & 2 \\\\\n",
|
||||
"\t16 & 16 & [0.0368842, 0.643704, 0.401421] & 3 \\\\\n",
|
||||
"\t17 & 17 & [0.525057, 0.61201] & 2 \\\\\n",
|
||||
"\t18 & 18 & [0.432577, 0.082207, 0.199058, 0.576082] & 4 \\\\\n",
|
||||
"\t19 & 19 & [0.218177, 0.362036, 0.204728, 0.932984] & 4 \\\\\n",
|
||||
"\t20 & 20 & [0.827263, 0.0992992, 0.6343] & 3 \\\\\n",
|
||||
"\t21 & 21 & [0.132715, 0.775194, 0.869237] & 3 \\\\\n",
|
||||
"\t22 & 22 & [0.0396356, 0.79041, 0.431188] & 3 \\\\\n",
|
||||
"\t23 & 23 & [0.137658, 0.60808, 0.255054] & 3 \\\\\n",
|
||||
"\t24 & 24 & [0.498734, 0.0940369, 0.52509] & 3 \\\\\n",
|
||||
"\t25 & 25 & [0.265511, 0.110096, 0.834362] & 3 \\\\\n",
|
||||
"\t26 & 26 & [0.633427, 0.337865, 0.112987] & 3 \\\\\n",
|
||||
"\t27 & 27 & [0.78299, 0.838042] & 2 \\\\\n",
|
||||
"\t28 & 28 & [0.0878598, 0.386568, 0.330579, 0.748041] & 4 \\\\\n",
|
||||
"\t29 & 29 & [0.265595, 0.291069, 0.612628] & 3 \\\\\n",
|
||||
"\t30 & 30 & [0.705766, 0.508363] & 2 \\\\\n",
|
||||
"\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
|
||||
"\\end{tabular}\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"\u001b[1m10000000×3 DataFrame\u001b[0m\n",
|
||||
"\u001b[1m Row \u001b[0m│\u001b[1m id \u001b[0m\u001b[1m pos \u001b[0m\u001b[1m jumps \u001b[0m\n",
|
||||
"\u001b[1m \u001b[0m│\u001b[90m Int64 \u001b[0m\u001b[90m Array… \u001b[0m\u001b[90m Int64 \u001b[0m\n",
|
||||
"──────────┼────────────────────────────────────────────────────\n",
|
||||
" 1 │ 1 [0.646691, 0.112486, 0.276021] 3\n",
|
||||
" 2 │ 2 [0.651664, 0.0566425, 0.842714] 3\n",
|
||||
" 3 │ 3 [0.950498, 0.96467] 2\n",
|
||||
" 4 │ 4 [0.945775, 0.789904] 2\n",
|
||||
" 5 │ 5 [0.82116, 0.0341601, 0.0945445, … 4\n",
|
||||
" 6 │ 6 [0.12781, 0.374187, 0.931115] 3\n",
|
||||
" 7 │ 7 [0.438939, 0.246862, 0.0118196, … 5\n",
|
||||
" 8 │ 8 [0.732, 0.299058] 2\n",
|
||||
" 9 │ 9 [0.449182, 0.875096] 2\n",
|
||||
" 10 │ 10 [0.0462887, 0.698356, 0.365109] 3\n",
|
||||
" 11 │ 11 [0.302478, 0.372575, 0.150508, 0… 5\n",
|
||||
" ⋮ │ ⋮ ⋮ ⋮\n",
|
||||
" 9999991 │ 9999991 [0.700468, 0.220524, 0.347931] 3\n",
|
||||
" 9999992 │ 9999992 [0.231368, 0.862016] 2\n",
|
||||
" 9999993 │ 9999993 [0.869351, 0.444795] 2\n",
|
||||
" 9999994 │ 9999994 [0.821356, 0.509054] 2\n",
|
||||
" 9999995 │ 9999995 [0.589245, 0.669708] 2\n",
|
||||
" 9999996 │ 9999996 [0.806262, 0.734397] 2\n",
|
||||
" 9999997 │ 9999997 [0.216506, 0.430571, 0.283787, 0… 4\n",
|
||||
" 9999998 │ 9999998 [0.0100723, 0.836315, 0.942299] 3\n",
|
||||
" 9999999 │ 9999999 [0.499669, 0.25214, 0.964065] 3\n",
|
||||
" 10000000 │ 10000000 [0.663339, 0.887989] 2\n",
|
||||
"\u001b[36m 9999979 rows omitted\u001b[0m"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"df"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table class=\"data-frame\"><thead><tr><th></th><th>first</th><th>last</th></tr><tr><th></th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>10,000,000 rows × 2 columns</p><tr><th>1</th><td>0.646691</td><td>0.276021</td></tr><tr><th>2</th><td>0.651664</td><td>0.842714</td></tr><tr><th>3</th><td>0.950498</td><td>0.96467</td></tr><tr><th>4</th><td>0.945775</td><td>0.789904</td></tr><tr><th>5</th><td>0.82116</td><td>0.314926</td></tr><tr><th>6</th><td>0.12781</td><td>0.931115</td></tr><tr><th>7</th><td>0.438939</td><td>0.496169</td></tr><tr><th>8</th><td>0.732</td><td>0.299058</td></tr><tr><th>9</th><td>0.449182</td><td>0.875096</td></tr><tr><th>10</th><td>0.0462887</td><td>0.365109</td></tr><tr><th>11</th><td>0.302478</td><td>0.283401</td></tr><tr><th>12</th><td>0.404953</td><td>0.658815</td></tr><tr><th>13</th><td>0.515627</td><td>0.59552</td></tr><tr><th>14</th><td>0.292462</td><td>0.61816</td></tr><tr><th>15</th><td>0.66426</td><td>0.753508</td></tr><tr><th>16</th><td>0.0368842</td><td>0.401421</td></tr><tr><th>17</th><td>0.525057</td><td>0.61201</td></tr><tr><th>18</th><td>0.432577</td><td>0.576082</td></tr><tr><th>19</th><td>0.218177</td><td>0.932984</td></tr><tr><th>20</th><td>0.827263</td><td>0.6343</td></tr><tr><th>21</th><td>0.132715</td><td>0.869237</td></tr><tr><th>22</th><td>0.0396356</td><td>0.431188</td></tr><tr><th>23</th><td>0.137658</td><td>0.255054</td></tr><tr><th>24</th><td>0.498734</td><td>0.52509</td></tr><tr><th>25</th><td>0.265511</td><td>0.834362</td></tr><tr><th>26</th><td>0.633427</td><td>0.112987</td></tr><tr><th>27</th><td>0.78299</td><td>0.838042</td></tr><tr><th>28</th><td>0.0878598</td><td>0.748041</td></tr><tr><th>29</th><td>0.265595</td><td>0.612628</td></tr><tr><th>30</th><td>0.705766</td><td>0.508363</td></tr><tr><th>⋮</th><td>⋮</td><td>⋮</td></tr></tbody></table>"
|
||||
],
|
||||
"text/latex": [
|
||||
"\\begin{tabular}{r|cc}\n",
|
||||
"\t& first & last\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t& Float64 & Float64\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t1 & 0.646691 & 0.276021 \\\\\n",
|
||||
"\t2 & 0.651664 & 0.842714 \\\\\n",
|
||||
"\t3 & 0.950498 & 0.96467 \\\\\n",
|
||||
"\t4 & 0.945775 & 0.789904 \\\\\n",
|
||||
"\t5 & 0.82116 & 0.314926 \\\\\n",
|
||||
"\t6 & 0.12781 & 0.931115 \\\\\n",
|
||||
"\t7 & 0.438939 & 0.496169 \\\\\n",
|
||||
"\t8 & 0.732 & 0.299058 \\\\\n",
|
||||
"\t9 & 0.449182 & 0.875096 \\\\\n",
|
||||
"\t10 & 0.0462887 & 0.365109 \\\\\n",
|
||||
"\t11 & 0.302478 & 0.283401 \\\\\n",
|
||||
"\t12 & 0.404953 & 0.658815 \\\\\n",
|
||||
"\t13 & 0.515627 & 0.59552 \\\\\n",
|
||||
"\t14 & 0.292462 & 0.61816 \\\\\n",
|
||||
"\t15 & 0.66426 & 0.753508 \\\\\n",
|
||||
"\t16 & 0.0368842 & 0.401421 \\\\\n",
|
||||
"\t17 & 0.525057 & 0.61201 \\\\\n",
|
||||
"\t18 & 0.432577 & 0.576082 \\\\\n",
|
||||
"\t19 & 0.218177 & 0.932984 \\\\\n",
|
||||
"\t20 & 0.827263 & 0.6343 \\\\\n",
|
||||
"\t21 & 0.132715 & 0.869237 \\\\\n",
|
||||
"\t22 & 0.0396356 & 0.431188 \\\\\n",
|
||||
"\t23 & 0.137658 & 0.255054 \\\\\n",
|
||||
"\t24 & 0.498734 & 0.52509 \\\\\n",
|
||||
"\t25 & 0.265511 & 0.834362 \\\\\n",
|
||||
"\t26 & 0.633427 & 0.112987 \\\\\n",
|
||||
"\t27 & 0.78299 & 0.838042 \\\\\n",
|
||||
"\t28 & 0.0878598 & 0.748041 \\\\\n",
|
||||
"\t29 & 0.265595 & 0.612628 \\\\\n",
|
||||
"\t30 & 0.705766 & 0.508363 \\\\\n",
|
||||
"\t$\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
|
||||
"\\end{tabular}\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"\u001b[1m10000000×2 DataFrame\u001b[0m\n",
|
||||
"\u001b[1m Row \u001b[0m│\u001b[1m first \u001b[0m\u001b[1m last \u001b[0m\n",
|
||||
"\u001b[1m \u001b[0m│\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\n",
|
||||
"──────────┼─────────────────────\n",
|
||||
" 1 │ 0.646691 0.276021\n",
|
||||
" 2 │ 0.651664 0.842714\n",
|
||||
" 3 │ 0.950498 0.96467\n",
|
||||
" 4 │ 0.945775 0.789904\n",
|
||||
" 5 │ 0.82116 0.314926\n",
|
||||
" 6 │ 0.12781 0.931115\n",
|
||||
" 7 │ 0.438939 0.496169\n",
|
||||
" 8 │ 0.732 0.299058\n",
|
||||
" 9 │ 0.449182 0.875096\n",
|
||||
" 10 │ 0.0462887 0.365109\n",
|
||||
" 11 │ 0.302478 0.283401\n",
|
||||
" ⋮ │ ⋮ ⋮\n",
|
||||
" 9999991 │ 0.700468 0.347931\n",
|
||||
" 9999992 │ 0.231368 0.862016\n",
|
||||
" 9999993 │ 0.869351 0.444795\n",
|
||||
" 9999994 │ 0.821356 0.509054\n",
|
||||
" 9999995 │ 0.589245 0.669708\n",
|
||||
" 9999996 │ 0.806262 0.734397\n",
|
||||
" 9999997 │ 0.216506 0.335015\n",
|
||||
" 9999998 │ 0.0100723 0.942299\n",
|
||||
" 9999999 │ 0.499669 0.964065\n",
|
||||
" 10000000 │ 0.663339 0.887989\n",
|
||||
"\u001b[36m 9999979 rows omitted\u001b[0m"
|
||||
]
|
||||
},
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"df_test = select(df, :pos => ByRow(first) => :first, :pos => ByRow(last) => :last)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAkIAAAGdCAYAAAD+JxxnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAprklEQVR4nO3df1DU953H8RcBWZGBLYqAG7kEbwiRYlOLDaLtaU4FU5HptHd6R7qjPYtmSCRUrNFJr9FMg/UX5hIbm3hezPmjZK7WXnoqheZ6ROLPELkLavSmMYoniInrgoYuBL/3R8bvZcUY4XbB5fN8zOwffPe9u5/vV2SfftldwyzLsgQAAGCgu/p7AQAAAP2FEAIAAMYihAAAgLEIIQAAYCxCCAAAGIsQAgAAxiKEAACAsQghAABgrIj+XsCd7tq1azp//rxiYmIUFhbW38sBAAC3wbIstbW1yeVy6a67Pv+8DyH0Bc6fP6/k5OT+XgYAAOiFxsZGjRw58nOvJ4S+QExMjKRPD2RsbGw/rwYAANyO1tZWJScn28/jn4cQ+gLXfx0WGxtLCAEAEGK+6GUtvFgaAAAYixACAADGIoQAAICxCCEAAGAsQggAABiLEAIAAMYihAAAgLEIIQAAYCxCCAAAGIsQAgAAxiKEAACAsQghAABgLEIIAAAYixACAADG6nEIvfnmm5o5c6ZcLpfCwsL0m9/8xu96y7K0fPlyuVwuRUVFafLkyTp27JjfjM/n08KFCxUfH6/o6Gjl5+fr3LlzfjMej0dut1tOp1NOp1Nut1uXL1/2mzl79qxmzpyp6OhoxcfHq7i4WB0dHX4z7777riZNmqSoqCjdfffdeuaZZ2RZVk93GyHk3qW7u10AALiZiJ7e4OrVq3rggQf0/e9/X9/97ne7Xb969WqVl5dry5Ytuu+++/TTn/5U06ZN08mTJxUTEyNJKikp0W9/+1tVVFRo2LBhKi0tVV5enurq6hQeHi5JKigo0Llz51RZWSlJmj9/vtxut377299Kkrq6ujRjxgwNHz5ctbW1+uijjzRnzhxZlqUXXnhBktTa2qpp06bpoYce0pEjR3Tq1CnNnTtX0dHRKi0t7d0RuwPc7In9g5/N6IeVIBTc+P3C9wr+v/iewkDS4xB6+OGH9fDDD9/0Osuy9Nxzz+mpp57Sd77zHUnSq6++qsTERO3YsUMLFiyQ1+vV5s2btXXrVk2dOlWStG3bNiUnJ+v3v/+9cnNzdeLECVVWVurgwYPKysqSJG3atEnZ2dk6efKk0tLSVFVVpePHj6uxsVEul0uStG7dOs2dO1fPPvusYmNjtX37dv3pT3/Sli1b5HA4lJGRoVOnTqm8vFyLFi1SWFhYrw4a+k8wI5Af7j0zUIJ8oPy5h+J+BGrNobjv/am//+7eaX9ePQ6hWzl9+rSam5uVk5Njb3M4HJo0aZL279+vBQsWqK6uTp2dnX4zLpdLGRkZ2r9/v3Jzc3XgwAE5nU47giRp/Pjxcjqd2r9/v9LS0nTgwAFlZGTYESRJubm58vl8qqur00MPPaQDBw5o0qRJcjgcfjPLli3TBx98oJSUlG774PP55PP57K9bW1sDdnxu5na+Ie70X+3091+q29GXP3D7cmagup3v+YFyPHqzr7dzm97+vezPOLmdNfd237/ofm92u97+PL6dNffmuPb2+PTln3soCmgINTc3S5ISExP9ticmJurMmTP2TGRkpOLi4rrNXL99c3OzEhISut1/QkKC38yNjxMXF6fIyEi/mXvvvbfb41y/7mYhtHLlSq1YseK29vdO1pdnTm5nJphP9L0JxUD9MA2UUFhPKJ55C9QxGwj/YJECFyfBmulL/b3mvvz5F6yfkQMllgIaQtfd+Csny7K+8NdQN87cbD4QM9dfKP1561m2bJkWLVpkf93a2qrk5ORbrj2Q+vIvXl8K5r9g+1N//zDtjTvteyxYsRJMff1nGqzHu9O+N28mFNZ4o0D9PRgIP/9CQUBDKCkpSdKnZ1tGjBhhb29pabHPxCQlJamjo0Mej8fvrFBLS4smTJhgz1y4cKHb/V+8eNHvfg4dOuR3vcfjUWdnp9/M9bNDn30cqftZq+scDoffr9JCxUD5hrzRQN2vUGTKn4Up+4k7H9+LfSOgnyOUkpKipKQkVVdX29s6OjpUU1NjR05mZqYGDRrkN9PU1KSGhgZ7Jjs7W16vV4cPH7ZnDh06JK/X6zfT0NCgpqYme6aqqkoOh0OZmZn2zJtvvun3lvqqqiq5XK5uvzIDEFr4iAQAgdDjELpy5Yrq6+tVX18v6dMXSNfX1+vs2bMKCwtTSUmJysrKtGvXLjU0NGju3LkaMmSICgoKJElOp1Pz5s1TaWmp3njjDR09elTf+973NGbMGPtdZKNHj9b06dNVWFiogwcP6uDBgyosLFReXp7S0tIkSTk5OUpPT5fb7dbRo0f1xhtvaPHixSosLFRsbKykT9+C73A4NHfuXDU0NGjXrl0qKyvjHWMAAEBSL3419vbbb+uhhx6yv77+epo5c+Zoy5YtWrJkidrb21VUVCSPx6OsrCxVVVXZnyEkSevXr1dERIRmzZql9vZ2TZkyRVu2bLE/Q0iStm/fruLiYvvdZfn5+dqwYYN9fXh4uHbv3q2ioiJNnDhRUVFRKigo0Nq1a+0Zp9Op6upqPfbYYxo3bpzi4uK0aNEiv9cAAQAAc4VZfMzyLbW2tsrpdMrr9dpnmgKJU/oAAJMF60Xht/v8zf81BgAAjEUIAQAAYxFCAADAWIQQAAAwFiEEAACMRQgBAABjEUIAAMBYhBAAADAWIQQAAIxFCAEAAGMRQgAAwFiEEAAAMBYhBAAAjEUIAQAAYxFCAADAWIQQAAAwFiEEAACMRQgBAABjEUIAAMBYhBAAADAWIQQAAIxFCAEAAGMRQgAAwFiEEAAAMBYhBAAAjEUIAQAAYxFCAADAWIQQAAAwFiEEAACMRQgBAABjEUIAAMBYhBAAADAWIQQAAIxFCAEAAGMRQgAAwFiEEAAAMBYhBAAAjEUIAQAAYxFCAADAWIQQAAAwFiEEAACMRQgBAABjEUIAAMBYhBAAADAWIQQAAIxFCAEAAGMRQgAAwFiEEAAAMBYhBAAAjEUIAQAAYxFCAADAWIQQAAAwFiEEAACMRQgBAABjEUIAAMBYhBAAADAWIQQAAIxFCAEAAGMFPIQ++eQT/fjHP1ZKSoqioqI0atQoPfPMM7p27Zo9Y1mWli9fLpfLpaioKE2ePFnHjh3zux+fz6eFCxcqPj5e0dHRys/P17lz5/xmPB6P3G63nE6nnE6n3G63Ll++7Ddz9uxZzZw5U9HR0YqPj1dxcbE6OjoCvdsAACAEBTyEVq1apV/84hfasGGDTpw4odWrV2vNmjV64YUX7JnVq1ervLxcGzZs0JEjR5SUlKRp06apra3NnikpKdGuXbtUUVGh2tpaXblyRXl5eerq6rJnCgoKVF9fr8rKSlVWVqq+vl5ut9u+vqurSzNmzNDVq1dVW1uriooK7dy5U6WlpYHebQAAEILCLMuyAnmHeXl5SkxM1ObNm+1t3/3udzVkyBBt3bpVlmXJ5XKppKRETz75pKRPz/4kJiZq1apVWrBggbxer4YPH66tW7dq9uzZkqTz588rOTlZe/bsUW5urk6cOKH09HQdPHhQWVlZkqSDBw8qOztb7733ntLS0rR3717l5eWpsbFRLpdLklRRUaG5c+eqpaVFsbGxX7g/ra2tcjqd8nq9tzXfU/cu3R3w+wQAIFR88LMZQbnf233+DvgZoW984xt64403dOrUKUnSf/7nf6q2tlbf+ta3JEmnT59Wc3OzcnJy7Ns4HA5NmjRJ+/fvlyTV1dWps7PTb8blcikjI8OeOXDggJxOpx1BkjR+/Hg5nU6/mYyMDDuCJCk3N1c+n091dXU3Xb/P51Nra6vfBQAADEwRgb7DJ598Ul6vV/fff7/Cw8PV1dWlZ599Vn/7t38rSWpubpYkJSYm+t0uMTFRZ86csWciIyMVFxfXbeb67Zubm5WQkNDt8RMSEvxmbnycuLg4RUZG2jM3WrlypVasWNHT3QYAACEo4GeEXnvtNW3btk07duzQO++8o1dffVVr167Vq6++6jcXFhbm97VlWd223ejGmZvN92bms5YtWyav12tfGhsbb7kmAAAQugJ+RuhHP/qRli5dqr/5m7+RJI0ZM0ZnzpzRypUrNWfOHCUlJUn69GzNiBEj7Nu1tLTYZ2+SkpLU0dEhj8fjd1aopaVFEyZMsGcuXLjQ7fEvXrzodz+HDh3yu97j8aizs7PbmaLrHA6HHA5Hb3cfAACEkICfEfr44491113+dxseHm6/fT4lJUVJSUmqrq62r+/o6FBNTY0dOZmZmRo0aJDfTFNTkxoaGuyZ7Oxseb1eHT582J45dOiQvF6v30xDQ4OamprsmaqqKjkcDmVmZgZ4zwEAQKgJ+BmhmTNn6tlnn9Wf/dmf6ctf/rKOHj2q8vJy/d3f/Z2kT39VVVJSorKyMqWmpio1NVVlZWUaMmSICgoKJElOp1Pz5s1TaWmphg0bpqFDh2rx4sUaM2aMpk6dKkkaPXq0pk+frsLCQr300kuSpPnz5ysvL09paWmSpJycHKWnp8vtdmvNmjW6dOmSFi9erMLCwqC8AwwAAISWgIfQCy+8oL//+79XUVGRWlpa5HK5tGDBAv3kJz+xZ5YsWaL29nYVFRXJ4/EoKytLVVVViomJsWfWr1+viIgIzZo1S+3t7ZoyZYq2bNmi8PBwe2b79u0qLi62312Wn5+vDRs22NeHh4dr9+7dKioq0sSJExUVFaWCggKtXbs20LsNAABCUMA/R2ig4XOEAAAIngH3OUIAAAChghACAADGIoQAAICxCCEAAGAsQggAABiLEAIAAMYihAAAgLEIIQAAYCxCCAAAGIsQAgAAxiKEAACAsQghAABgLEIIAAAYixACAADLine truncated
|
||||
"text/plain": [
|
||||
"Figure(PyObject <Figure size 640x480 with 1 Axes>)"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"hist(df_test.first, 100);"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"So far all looks good. But let us look at the distribution of the last dawn random number:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAkIAAAGdCAYAAAD+JxxnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA4YElEQVR4nO3df1RU953/8dcEZIIcuUEQyCQkMXsMkWCTLDaI9lvMqmAr0my3q13SqXQtMQcjoUAS7Y9UPSloNJhWG2vctHSNWXK2Kdm4KoGmjUoVVCKtqNE00SAFxMRxUGOA4P3+kcPdjBgVM6Bwn49z5o+59z13Pvce23nl/fnci8M0TVMAAAA2dN3VHgAAAMDVQhACAAC2RRACAAC2RRACAAC2RRACAAC2RRACAAC2RRACAAC2RRACAAC2FXi1B3CtO3funJqamjRs2DA5HI6rPRwAAHAZTNPUqVOn5HK5dN11n9/3IQhdQlNTk2JiYq72MAAAwBU4evSobr755s/dTxC6hGHDhkn69EKGhoZe5dEAAIDL0dbWppiYGOt3/PMQhC6hezosNDSUIAQAwABzqWUtLJYGAAC2RRACAAC2RRACAAC2RRACAAC2RRACAAC2RRACAAC2RRACAAC2RRACAAC2RRACAAC2RRACAAC2RRACAAC2RRACAAC2RRACAAC2RRACAAC2FXi1BwAAAOzjtvkbfd4fWTLtKo3kU3SEAACAbRGEAACAbTE1BgAA+sT502DXIoIQAADwi4EQfM5HEAIAAL02EEPPhbBGCAAA2BYdIQAAcEmDpQN0PjpCAADAtugIAQAAH4O1+3MhdIQAAIBtEYQAAIBtMTUGAIDN2Wkq7Hx0hAAAgG0RhAAAgG0xNQYAgI3YeRrsQghCAAAMYgSfi2NqDAAA2BZBCAAA2Favg9DWrVs1ffp0uVwuORwOvfrqqz1qDhw4oPT0dBmGoWHDhmncuHFqaGiw9re3t2vevHmKiIhQSEiI0tPT1djY6HMMj8cjt9stwzBkGIbcbrdOnjzpU9PQ0KDp06crJCREERERysnJUUdHh0/N3r17lZycrODgYN10001avHixTNPs7WkDAHDNu23+xh4vXFyvg9CZM2d09913a9WqVRfc/+677+orX/mK7rzzTr355pv6y1/+op/85Ce6/vrrrZrc3FyVlZWptLRUVVVVOn36tNLS0tTV1WXVZGRkqK6uTuXl5SovL1ddXZ3cbre1v6urS9OmTdOZM2dUVVWl0tJSvfLKK8rPz7dq2traNGXKFLlcLu3atUsrV67U8uXLVVxc3NvTBgAAg5DD/ALtEYfDobKyMj3wwAPWtm9/+9saMmSI1q1bd8HPeL1ejRgxQuvWrdPMmTMlSU1NTYqJidGmTZuUmpqqAwcOKC4uTtXV1UpMTJQkVVdXKykpSW+//bZiY2O1efNmpaWl6ejRo3K5XJKk0tJSZWZmqrW1VaGhoVq9erUWLFigY8eOyel0SpKWLFmilStXqrGxUQ6H45Ln2NbWJsMw5PV6FRoaeqWXCgCAPjcQO0BHlkzrk+Ne7u+3X9cInTt3Ths3btQdd9yh1NRURUZGKjEx0Wf6rLa2Vp2dnUpJSbG2uVwuxcfHa/v27ZKkHTt2yDAMKwRJ0rhx42QYhk9NfHy8FYIkKTU1Ve3t7aqtrbVqkpOTrRDUXdPU1KQjR45c8Bza29vV1tbm8wIA4FrENNgX59cg1NraqtOnT2vJkiWaOnWqKioq9M///M/65je/qS1btkiSWlpaFBQUpLCwMJ/PRkVFqaWlxaqJjIzscfzIyEifmqioKJ/9YWFhCgoKumhN9/vumvMVFRVZ65IMw1BMTExvLwMAABgg/N4RkqRvfOMb+sEPfqB77rlH8+fPV1pamn71q19d9LOmafpMVV1o2sofNd0zgZ83LbZgwQJ5vV7rdfTo0YuOGwAADFx+faBiRESEAgMDFRcX57N99OjRqqqqkiRFR0ero6NDHo/HpyvU2tqq8ePHWzXHjh3rcfzjx49bHZ3o6GjV1NT47Pd4POrs7PSpOb/z09raKkk9OkXdnE6nz1QaAADXAqa++oZfO0JBQUH68pe/rIMHD/psP3TokG699VZJUkJCgoYMGaLKykprf3Nzs+rr660glJSUJK/Xq507d1o1NTU18nq9PjX19fVqbm62aioqKuR0OpWQkGDVbN261eeW+oqKCrlcLt12223+PHUAAPyK9T/9o9cdodOnT+tvf/ub9f7w4cOqq6vT8OHDdcstt+ixxx7TzJkz9dWvflX333+/ysvLtWHDBr355puSJMMwNHv2bOXn5ys8PFzDhw9XQUGBxowZo8mTJ0v6tIM0depUZWVlac2aNZKkhx56SGlpaYqNjZUkpaSkKC4uTm63W8uWLdOJEydUUFCgrKwsa3V4RkaGFi1apMzMTP3whz/UO++8o8LCQj355JOXdccYAAAY3Hp9+/ybb76p+++/v8f2WbNmqaSkRJL061//WkVFRWpsbFRsbKwWLVqkb3zjG1btxx9/rMcee0wvvfSSzp49q0mTJum5557zWZh84sQJ5eTk6LXXXpMkpaena9WqVbrhhhusmoaGBmVnZ+uPf/yjgoODlZGRoeXLl/tMbe3du1dz587Vzp07FRYWpocffrhXQYjb5wEAfc3OHZ+rffv8F3qOkB0QhAAAfY0g5H9X5TlCAAAAA4lf7xoDAACXZucO0LWGjhAAALAtghAAALAtpsYAAOhDTINd2+gIAQAA26IjBACAH9EBGljoCAEAANsiCAEAANtiagwAgCvENNjAR0cIAADYFh0hAAAuEx2gwYcgBADABRB67IGpMQAAYFsEIQAAYFsEIQAAYFusEQIAQKwJsiuCEADAdgg96MbUGAAAsC2CEAAAsC2CEAAAsC3WCAEABj3WBOHz0BECAAC2RRACAAC2RRACAAC2xRohAMCAxvoffBF0hAAAgG0RhAAAgG0RhAAAgG2xRggAMGCwHgj+1uuO0NatWzV9+nS5XC45HA69+uqrn1s7Z84cORwOPfvssz7b29vbNW/ePEVERCgkJETp6elqbGz0qfF4PHK73TIMQ4ZhyO126+TJkz41DQ0Nmj59ukJCQhQREaGcnBx1dHT41Ozdu1fJyckKDg7WTTfdpMWLF8s0zd6eNgAAGIR6HYTOnDmju+++W6tWrbpo3auvvqqamhq5XK4e+3Jzc1VWVqbS0lJVVVXp9OnTSktLU1dXl1WTkZGhuro6lZeXq7y8XHV1dXK73db+rq4uTZs2TWfOnFFVVZVKS0v1yiuvKD8/36ppa2vTlClT5HK5tGvXLq1cuVLLly9XcXFxb08bAAAMQr2eGvva176mr33taxet+fvf/65HHnlEr7/+uqZNm+azz+v16oUXXtC6des0efJkSdKLL76omJgY/eEPf1BqaqoOHDig8vJyVVdXKzExUZK0du1aJSUl6eDBg4qNjVVFRYX279+vo0ePWmHrmWeeUWZmpn72s58pNDRU69ev18cff6ySkhI5nU7Fx8fr0KFDKi4uVl5enhwOR29PHwDQj5gKQ1/z+2Lpc+fOye1267HHHtNdd93VY39tba06OzuVkpJibXO5XIqPj9f27dslSTt27JBhGFYIkqRx48bJMAyfmvj4eJ+OU2pqqtrb21VbW2vVJCcny+l0+tQ0NTXpyJEjFxx/e3u72trafF4AAGBw8vti6aVLlyowMFA5OTkX3N/S0qKgoCCFhYX5bI+KilJLS4tVExkZ2eOzkZGRPjVRUVE++8PCwhQUFORTc9ttt/X4nu59I0eO7PEdRUVFWrRo0WWcKQDAn+j+4Grwa0eotrZWP//5z1VSUtLraSfTNH0+c6HP+6Ome6H0541vwYIF8nq91uvo0aO9Og8AADBw+LUjtG3bNrW2tuqWW26xtnV1dSk/P1/PPvusjhw5oujoaHV0dMjj8fh0hVpbWzV+/HhJUnR0tI4dO9bj+MePH7c6OtHR0aqpqfHZ7/F41NnZ6VPT3R367PdI6tFN6uZ0On2m0gAAfYMOEK4Ffu0Iud1u/fWvf1VdXZ31crlceuyxx/T6669LkhISEjRkyBBVVlZan2tublZ9fb0VhJKSkuT1erVz506rpqamRl6v16emvr5ezc3NVk1FRYWcTqcSEhKsmq1bt/rcUl9RUSGXy9VjygwAANhPrztCp0+f1t/+9jfr/eHDh1VXV6fhw4frlltuUXh4uE/9kCFDFB0drdjYWEmSYRiaPXu28vPzFR4eruHDh6ugoEBjxoyx7iIbPXq0pk6dqqysLK1Zs0aS9NBDDyktLc06TkpKiuLi4uR2u7Vs2TKdOHFCBQUFysrKUmhoqKRPb8FftGiRMjMz9cMf/lDvvPOOCgsL9eSTT3LHGAAA6H0Q2r17t+6//37rfV5eniRp1qxZKikpuaxjrFixQoGBgZoxY4bOnj2rSZMmqaSkRAEBAVbN+vXrlZOTY91dlp6e7vPsooCAAG3cuFHZ2dmaMGGCgoODlZGRoeXLl1s1hmGosrJSc+fO1dixYxUWFqa8vDxrzAAAwN4cJo9Zvqi2tjYZhiGv12t1mgAAXxxrhCBJR5ZMu3TRFbjc32/+1hgAoM8RenCt4q/PAwAA26IjBADwOzpAGCjoCAEAANuiIwQA+ELo/mAgoyMEAABsiyAEAABsi6kxAECvMBWGwYQgBAD4XIQeDHZMjQEAANsiCAEAANsiCAEAANsiCAEAANtisTQAwMLiaNgNHSEAAGBbdIQAwKbo/gB0hAAAgI0RhAAAgG0xNQYANsFUGNATHSEAAGBbBCEAAGBbBCEAAGBbrBECgEGI9UDA5aEjBAAAbIuOEAAMAnSAgCtDRwgAANgWQQgAANgWQQgAANgWa4QAYIBhPRDgP3SEAACAbfU6CG3dulXTp0+Xy+WSw+HQq6++au3r7OzUE088oTFjxigkJEQul0vf/e531dTU5HOM9vZLine truncated
|
||||
"text/plain": [
|
||||
"Figure(PyObject <Figure size 640x480 with 1 Axes>)"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"hist(df_test.last, 100);"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"So - is the `rand()` function broken for the last generated random number in each sequence or something else has made the distribution stop being uniform?"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Julia 1.6.1",
|
||||
"language": "julia",
|
||||
"name": "julia-1.6"
|
||||
},
|
||||
"language_info": {
|
||||
"file_extension": ".jl",
|
||||
"mimetype": "application/julia",
|
||||
"name": "julia",
|
||||
"version": "1.6.1"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,479 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Final examples\n",
|
||||
"\n",
|
||||
"### Bogumił Kamiński"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let us wrap up our tutorial with examples of joining and reshaping data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Joining and reshaping data frames"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"using DataFrames\n",
|
||||
"using CSV\n",
|
||||
"using Pipe\n",
|
||||
"using Unitful\n",
|
||||
"using Dates"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Load the weather forecast data from two cities from Poland."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table class=\"data-frame\"><thead><tr><th></th><th>city</th><th>date</th><th>rainfall</th></tr><tr><th></th><th>String</th><th>Date</th><th>Float64</th></tr></thead><tbody><p>10 rows × 3 columns</p><tr><th>1</th><td>Olecko</td><td>2020-11-16</td><td>2.9</td></tr><tr><th>2</th><td>Olecko</td><td>2020-11-17</td><td>4.1</td></tr><tr><th>3</th><td>Olecko</td><td>2020-11-19</td><td>4.3</td></tr><tr><th>4</th><td>Olecko</td><td>2020-11-20</td><td>2.0</td></tr><tr><th>5</th><td>Olecko</td><td>2020-11-21</td><td>0.6</td></tr><tr><th>6</th><td>Olecko</td><td>2020-11-22</td><td>1.0</td></tr><tr><th>7</th><td>Ełk</td><td>2020-11-16</td><td>3.9</td></tr><tr><th>8</th><td>Ełk</td><td>2020-11-19</td><td>1.2</td></tr><tr><th>9</th><td>Ełk</td><td>2020-11-20</td><td>2.0</td></tr><tr><th>10</th><td>Ełk</td><td>2020-11-22</td><td>2.0</td></tr></tbody></table>"
|
||||
],
|
||||
"text/latex": [
|
||||
"\\begin{tabular}{r|ccc}\n",
|
||||
"\t& city & date & rainfall\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t& String & Date & Float64\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t1 & Olecko & 2020-11-16 & 2.9 \\\\\n",
|
||||
"\t2 & Olecko & 2020-11-17 & 4.1 \\\\\n",
|
||||
"\t3 & Olecko & 2020-11-19 & 4.3 \\\\\n",
|
||||
"\t4 & Olecko & 2020-11-20 & 2.0 \\\\\n",
|
||||
"\t5 & Olecko & 2020-11-21 & 0.6 \\\\\n",
|
||||
"\t6 & Olecko & 2020-11-22 & 1.0 \\\\\n",
|
||||
"\t7 & Ełk & 2020-11-16 & 3.9 \\\\\n",
|
||||
"\t8 & Ełk & 2020-11-19 & 1.2 \\\\\n",
|
||||
"\t9 & Ełk & 2020-11-20 & 2.0 \\\\\n",
|
||||
"\t10 & Ełk & 2020-11-22 & 2.0 \\\\\n",
|
||||
"\\end{tabular}\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"\u001b[1m10×3 DataFrame\u001b[0m\n",
|
||||
"\u001b[1m Row \u001b[0m│\u001b[1m city \u001b[0m\u001b[1m date \u001b[0m\u001b[1m rainfall \u001b[0m\n",
|
||||
"\u001b[1m \u001b[0m│\u001b[90m String \u001b[0m\u001b[90m Date \u001b[0m\u001b[90m Float64 \u001b[0m\n",
|
||||
"─────┼──────────────────────────────\n",
|
||||
" 1 │ Olecko 2020-11-16 2.9\n",
|
||||
" 2 │ Olecko 2020-11-17 4.1\n",
|
||||
" 3 │ Olecko 2020-11-19 4.3\n",
|
||||
" 4 │ Olecko 2020-11-20 2.0\n",
|
||||
" 5 │ Olecko 2020-11-21 0.6\n",
|
||||
" 6 │ Olecko 2020-11-22 1.0\n",
|
||||
" 7 │ Ełk 2020-11-16 3.9\n",
|
||||
" 8 │ Ełk 2020-11-19 1.2\n",
|
||||
" 9 │ Ełk 2020-11-20 2.0\n",
|
||||
" 10 │ Ełk 2020-11-22 2.0"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"rainfall_long = CSV.File(\"data/rainfall_forecast.csv\") |> DataFrame"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Note that we collect rainfall information, so it would be nice to add units to the measured values. This is not a problem with `Unitful.jl`. We take advantage of the fact that `DataFrame` can store vectors of any Julia objects."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table class=\"data-frame\"><thead><tr><th></th><th>city</th><th>date</th><th>rainfall</th></tr><tr><th></th><th>String</th><th>Date</th><th>Quantit…</th></tr></thead><tbody><p>10 rows × 3 columns</p><tr><th>1</th><td>Olecko</td><td>2020-11-16</td><td>2.9 mm</td></tr><tr><th>2</th><td>Olecko</td><td>2020-11-17</td><td>4.1 mm</td></tr><tr><th>3</th><td>Olecko</td><td>2020-11-19</td><td>4.3 mm</td></tr><tr><th>4</th><td>Olecko</td><td>2020-11-20</td><td>2.0 mm</td></tr><tr><th>5</th><td>Olecko</td><td>2020-11-21</td><td>0.6 mm</td></tr><tr><th>6</th><td>Olecko</td><td>2020-11-22</td><td>1.0 mm</td></tr><tr><th>7</th><td>Ełk</td><td>2020-11-16</td><td>3.9 mm</td></tr><tr><th>8</th><td>Ełk</td><td>2020-11-19</td><td>1.2 mm</td></tr><tr><th>9</th><td>Ełk</td><td>2020-11-20</td><td>2.0 mm</td></tr><tr><th>10</th><td>Ełk</td><td>2020-11-22</td><td>2.0 mm</td></tr></tbody></table>"
|
||||
],
|
||||
"text/latex": [
|
||||
"\\begin{tabular}{r|ccc}\n",
|
||||
"\t& city & date & rainfall\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t& String & Date & Quantit…\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t1 & Olecko & 2020-11-16 & 2.9 mm \\\\\n",
|
||||
"\t2 & Olecko & 2020-11-17 & 4.1 mm \\\\\n",
|
||||
"\t3 & Olecko & 2020-11-19 & 4.3 mm \\\\\n",
|
||||
"\t4 & Olecko & 2020-11-20 & 2.0 mm \\\\\n",
|
||||
"\t5 & Olecko & 2020-11-21 & 0.6 mm \\\\\n",
|
||||
"\t6 & Olecko & 2020-11-22 & 1.0 mm \\\\\n",
|
||||
"\t7 & Ełk & 2020-11-16 & 3.9 mm \\\\\n",
|
||||
"\t8 & Ełk & 2020-11-19 & 1.2 mm \\\\\n",
|
||||
"\t9 & Ełk & 2020-11-20 & 2.0 mm \\\\\n",
|
||||
"\t10 & Ełk & 2020-11-22 & 2.0 mm \\\\\n",
|
||||
"\\end{tabular}\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"\u001b[1m10×3 DataFrame\u001b[0m\n",
|
||||
"\u001b[1m Row \u001b[0m│\u001b[1m city \u001b[0m\u001b[1m date \u001b[0m\u001b[1m rainfall \u001b[0m\n",
|
||||
"\u001b[1m \u001b[0m│\u001b[90m String \u001b[0m\u001b[90m Date \u001b[0m\u001b[90m Quantity… \u001b[0m\n",
|
||||
"─────┼───────────────────────────────\n",
|
||||
" 1 │ Olecko 2020-11-16 2.9 mm\n",
|
||||
" 2 │ Olecko 2020-11-17 4.1 mm\n",
|
||||
" 3 │ Olecko 2020-11-19 4.3 mm\n",
|
||||
" 4 │ Olecko 2020-11-20 2.0 mm\n",
|
||||
" 5 │ Olecko 2020-11-21 0.6 mm\n",
|
||||
" 6 │ Olecko 2020-11-22 1.0 mm\n",
|
||||
" 7 │ Ełk 2020-11-16 3.9 mm\n",
|
||||
" 8 │ Ełk 2020-11-19 1.2 mm\n",
|
||||
" 9 │ Ełk 2020-11-20 2.0 mm\n",
|
||||
" 10 │ Ełk 2020-11-22 2.0 mm"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"transform!(rainfall_long, :rainfall => x -> x .* u\"mm\", renamecols=false)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"With `renamecols=false` we left the name of the transformed column unchanged when we did an in-place update of the data frame using the `transform!` function."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"It would be nice to see the data in a wide format, so that each city is represented by a single column. We can achieve this using the `unstack` function:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table class=\"data-frame\"><thead><tr><th></th><th>date</th><th>Olecko</th><th>Ełk</th></tr><tr><th></th><th>Date</th><th>Quantit…?</th><th>Quantit…?</th></tr></thead><tbody><p>6 rows × 3 columns</p><tr><th>1</th><td>2020-11-16</td><td>2.9 mm</td><td>3.9 mm</td></tr><tr><th>2</th><td>2020-11-17</td><td>4.1 mm</td><td><em>missing</em></td></tr><tr><th>3</th><td>2020-11-19</td><td>4.3 mm</td><td>1.2 mm</td></tr><tr><th>4</th><td>2020-11-20</td><td>2.0 mm</td><td>2.0 mm</td></tr><tr><th>5</th><td>2020-11-21</td><td>0.6 mm</td><td><em>missing</em></td></tr><tr><th>6</th><td>2020-11-22</td><td>1.0 mm</td><td>2.0 mm</td></tr></tbody></table>"
|
||||
],
|
||||
"text/latex": [
|
||||
"\\begin{tabular}{r|ccc}\n",
|
||||
"\t& date & Olecko & Ełk\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t& Date & Quantit…? & Quantit…?\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t1 & 2020-11-16 & 2.9 mm & 3.9 mm \\\\\n",
|
||||
"\t2 & 2020-11-17 & 4.1 mm & \\emph{missing} \\\\\n",
|
||||
"\t3 & 2020-11-19 & 4.3 mm & 1.2 mm \\\\\n",
|
||||
"\t4 & 2020-11-20 & 2.0 mm & 2.0 mm \\\\\n",
|
||||
"\t5 & 2020-11-21 & 0.6 mm & \\emph{missing} \\\\\n",
|
||||
"\t6 & 2020-11-22 & 1.0 mm & 2.0 mm \\\\\n",
|
||||
"\\end{tabular}\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"\u001b[1m6×3 DataFrame\u001b[0m\n",
|
||||
"\u001b[1m Row \u001b[0m│\u001b[1m date \u001b[0m\u001b[1m Olecko \u001b[0m\u001b[1m Ełk \u001b[0m\n",
|
||||
"\u001b[1m \u001b[0m│\u001b[90m Date \u001b[0m\u001b[90m Quantity…? \u001b[0m\u001b[90m Quantity…? \u001b[0m\n",
|
||||
"─────┼──────────────────────────────────────\n",
|
||||
" 1 │ 2020-11-16 2.9 mm 3.9 mm\n",
|
||||
" 2 │ 2020-11-17 4.1 mm \u001b[90m missing \u001b[0m\n",
|
||||
" 3 │ 2020-11-19 4.3 mm 1.2 mm\n",
|
||||
" 4 │ 2020-11-20 2.0 mm 2.0 mm\n",
|
||||
" 5 │ 2020-11-21 0.6 mm \u001b[90m missing \u001b[0m\n",
|
||||
" 6 │ 2020-11-22 1.0 mm 2.0 mm"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"rainfall_wide = unstack(rainfall_long, :date, :city, :rainfall)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can see that the \"gaps\" in the rainfall information for `\"Ełk\"` column got automatically filled by `missing`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"There is also a `stack` function that does the reverse: transforms a data frame from wide to long format."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Also note that one of the cities is `\"Ełk\"`, which has a non standard character `ł` in its name. It is not a problem with `DataFrames.jl`. Let us e.g. extract this column as an exercise:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"6-element Vector{Union{Missing, Quantity{Float64, 𝐋, Unitful.FreeUnits{(mm,), 𝐋, nothing}}}}:\n",
|
||||
" 3.9 mm\n",
|
||||
" missing\n",
|
||||
" 1.2 mm\n",
|
||||
" 2.0 mm\n",
|
||||
" missing\n",
|
||||
" 2.0 mm"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"rainfall_wide.Ełk"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"6-element Vector{Union{Missing, Quantity{Float64, 𝐋, Unitful.FreeUnits{(mm,), 𝐋, nothing}}}}:\n",
|
||||
" 3.9 mm\n",
|
||||
" missing\n",
|
||||
" 1.2 mm\n",
|
||||
" 2.0 mm\n",
|
||||
" missing\n",
|
||||
" 2.0 mm"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"rainfall_wide.\"Ełk\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"When we read the data, we note that still there are gaps in the passed information --- one of the days is missing as there is no forecasted rainfall for it.\n",
|
||||
"\n",
|
||||
"It would be nice to have information for all days in the considered period. Here is the way to do it:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table class=\"data-frame\"><thead><tr><th></th><th>date</th></tr><tr><th></th><th>Date</th></tr></thead><tbody><p>7 rows × 1 columns</p><tr><th>1</th><td>2020-11-16</td></tr><tr><th>2</th><td>2020-11-17</td></tr><tr><th>3</th><td>2020-11-18</td></tr><tr><th>4</th><td>2020-11-19</td></tr><tr><th>5</th><td>2020-11-20</td></tr><tr><th>6</th><td>2020-11-21</td></tr><tr><th>7</th><td>2020-11-22</td></tr></tbody></table>"
|
||||
],
|
||||
"text/latex": [
|
||||
"\\begin{tabular}{r|c}\n",
|
||||
"\t& date\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t& Date\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t1 & 2020-11-16 \\\\\n",
|
||||
"\t2 & 2020-11-17 \\\\\n",
|
||||
"\t3 & 2020-11-18 \\\\\n",
|
||||
"\t4 & 2020-11-19 \\\\\n",
|
||||
"\t5 & 2020-11-20 \\\\\n",
|
||||
"\t6 & 2020-11-21 \\\\\n",
|
||||
"\t7 & 2020-11-22 \\\\\n",
|
||||
"\\end{tabular}\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"\u001b[1m7×1 DataFrame\u001b[0m\n",
|
||||
"\u001b[1m Row \u001b[0m│\u001b[1m date \u001b[0m\n",
|
||||
"\u001b[1m \u001b[0m│\u001b[90m Date \u001b[0m\n",
|
||||
"─────┼────────────\n",
|
||||
" 1 │ 2020-11-16\n",
|
||||
" 2 │ 2020-11-17\n",
|
||||
" 3 │ 2020-11-18\n",
|
||||
" 4 │ 2020-11-19\n",
|
||||
" 5 │ 2020-11-20\n",
|
||||
" 6 │ 2020-11-21\n",
|
||||
" 7 │ 2020-11-22"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"all_days = DataFrame(date=Date.(2020,11, 16:22))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<table class=\"data-frame\"><thead><tr><th></th><th>date</th><th>Olecko</th><th>Ełk</th></tr><tr><th></th><th>Date</th><th>Quantit…</th><th>Quantit…</th></tr></thead><tbody><p>7 rows × 3 columns</p><tr><th>1</th><td>2020-11-16</td><td>2.9 mm</td><td>3.9 mm</td></tr><tr><th>2</th><td>2020-11-17</td><td>4.1 mm</td><td>0.0 mm</td></tr><tr><th>3</th><td>2020-11-19</td><td>4.3 mm</td><td>1.2 mm</td></tr><tr><th>4</th><td>2020-11-20</td><td>2.0 mm</td><td>2.0 mm</td></tr><tr><th>5</th><td>2020-11-21</td><td>0.6 mm</td><td>0.0 mm</td></tr><tr><th>6</th><td>2020-11-22</td><td>1.0 mm</td><td>2.0 mm</td></tr><tr><th>7</th><td>2020-11-18</td><td>0.0 mm</td><td>0.0 mm</td></tr></tbody></table>"
|
||||
],
|
||||
"text/latex": [
|
||||
"\\begin{tabular}{r|ccc}\n",
|
||||
"\t& date & Olecko & Ełk\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t& Date & Quantit… & Quantit…\\\\\n",
|
||||
"\t\\hline\n",
|
||||
"\t1 & 2020-11-16 & 2.9 mm & 3.9 mm \\\\\n",
|
||||
"\t2 & 2020-11-17 & 4.1 mm & 0.0 mm \\\\\n",
|
||||
"\t3 & 2020-11-19 & 4.3 mm & 1.2 mm \\\\\n",
|
||||
"\t4 & 2020-11-20 & 2.0 mm & 2.0 mm \\\\\n",
|
||||
"\t5 & 2020-11-21 & 0.6 mm & 0.0 mm \\\\\n",
|
||||
"\t6 & 2020-11-22 & 1.0 mm & 2.0 mm \\\\\n",
|
||||
"\t7 & 2020-11-18 & 0.0 mm & 0.0 mm \\\\\n",
|
||||
"\\end{tabular}\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"\u001b[1m7×3 DataFrame\u001b[0m\n",
|
||||
"\u001b[1m Row \u001b[0m│\u001b[1m date \u001b[0m\u001b[1m Olecko \u001b[0m\u001b[1m Ełk \u001b[0m\n",
|
||||
"\u001b[1m \u001b[0m│\u001b[90m Date \u001b[0m\u001b[90m Quantity… \u001b[0m\u001b[90m Quantity… \u001b[0m\n",
|
||||
"─────┼──────────────────────────────────\n",
|
||||
" 1 │ 2020-11-16 2.9 mm 3.9 mm\n",
|
||||
" 2 │ 2020-11-17 4.1 mm 0.0 mm\n",
|
||||
" 3 │ 2020-11-19 4.3 mm 1.2 mm\n",
|
||||
" 4 │ 2020-11-20 2.0 mm 2.0 mm\n",
|
||||
" 5 │ 2020-11-21 0.6 mm 0.0 mm\n",
|
||||
" 6 │ 2020-11-22 1.0 mm 2.0 mm\n",
|
||||
" 7 │ 2020-11-18 0.0 mm 0.0 mm"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"@pipe leftjoin(all_days, rainfall_wide, on=:date) |>\n",
|
||||
" coalesce.(_, 0.0u\"mm\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Note that we additionally used a broadcasted `coalesce` operation on the whole data frame returned from `leftjoin` to replace all `missing` values by `0.0u\"mm\"` in it, as in this case `missing` meant that there is no rain forecasted for that day.\n",
|
||||
"\n",
|
||||
"It was safe to do here, as we knew that `:date` column does not contain missings. In particular note that `leftjoin` would error by default if we tried to perfrom join on a column that contains `missing` values (use `matchmissing` keyword argument in joins to change this behavior)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Conclusions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Before we finish let us summarize the major functions that `DataFrames.jl` provides:\n",
|
||||
"1. data frame is a matrix-like data structure. You can index it just like a matrix. The differences are\n",
|
||||
" - you can use strings or `Symbol`s to select columns\n",
|
||||
" - if you select rows with `!` it selects you whole column of a data frame and passes it to you without copying\n",
|
||||
"2. You can quickly summarize the contents of a data frame using the `describe` function\n",
|
||||
"3. You can add rows to a data frame in-place using `push!` (similarly `append!` allows you to add multiple rows at the same time) (also `repeat`/`repeat!`, `hcat` and `vcat` are provided)\n",
|
||||
"4. You can work on a grouped data frame that is created using the `groupby` function. It is a view and works as-if you have created a lookup index to a data frame.\n",
|
||||
"5. There are `select`/`select!`/`transform`/`transform!`/`combine` functions that allow you to quickly transform/aggregate columns of a data frame or grouped data frame; there is also `mapcols`/`mapcols!` functions for quick aggregation of columns of a data frame\n",
|
||||
"6. You can filter rows of a data frame using `filter` and `filter!` functions (also `subset` and `subset!` starting from version 1.0)\n",
|
||||
"7. Use `sort` and `sort!` functions to sort data frames\n",
|
||||
"8. You can join multiple data frames using `innerjoin`, `outerjoin`, `leftjoin`, `rightjoin`, `semijoin`, `antijoin`, and `crossjoin` functions (they work as you would expect them if you know SQL)\n",
|
||||
"9. If you want to iterate rows or columns of a data frame use `eachrow` and `eachcol` functions (we have not discussed them, but they work exactly like in Julia Base)\n",
|
||||
"10. You can change names of columns in a data frame using `rename` and `rename!` functions; to get names of columns of a data frame use `names` (strings) or `propertynames` (`Symbol`s)\n",
|
||||
"11. To get number of rows and columns of a data frame use `nrow` and `ncol` functions\n",
|
||||
"12. To flatten nested columns of a data frame use `flatten`\n",
|
||||
"13. You can easily allow/disallow missing values in columns of a data frame using `allowmising`/`allowmissing!`/`disallowmising`/`disallowmissing!` functions\n",
|
||||
"14. You can drop rows with missing data with `dropmissing`/`dropmissing!` functions\n",
|
||||
"15. You can switch between [long and wide](https://en.wikipedia.org/wiki/Wide_and_narrow_data) representation of a data frame using `stack` and `unstack`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Additionally we have covered `freqtable` from FreqTables.jl, `@pipe` from Pipe.jl, and `lm` from GLM.jl packages that are often useful when wrangling data.\n",
|
||||
"\n",
|
||||
"You can use many formats to store and read data frames, we have discussed CSV.jl and Arrow.jl packages that provide such functionality.\n",
|
||||
"\n",
|
||||
"Finally we have shown how to integrate DataFrames.jl with plotting using PyPlot.jl and Unitful.jl."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Of course this course was just an introduction.\n",
|
||||
"\n",
|
||||
"You can find reviews of functionality of DataFrames.jl in:\n",
|
||||
"* an official manual at https://juliadata.github.io/DataFrames.jl/stable/\n",
|
||||
"* a tutorial going through all functionalities of DataFrames.jl at https://github.com/bkamins/Julia-DataFrames-Tutorial\n",
|
||||
"* documentation strings of the respective funcions"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Julia 1.6.1",
|
||||
"language": "julia",
|
||||
"name": "julia-1.6"
|
||||
},
|
||||
"language_info": {
|
||||
"file_extension": ".jl",
|
||||
"mimetype": "application/julia",
|
||||
"name": "julia",
|
||||
"version": "1.6.1"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
Reference in new issue
Block a user