JuliaForDataAnalysis/exercises/exercises10.md

Julia for Data Analysis

Bogumił Kamiński, Daniel Kaszyński

Chapter 10

Problems

Exercise 1

Generate a random matrix mat having size 5x4 and all elements drawn independently and uniformly from the [0,1[ interval. Create a data frame using data from this matrix using auto-generated column names.

Solution

julia> using DataFrames

julia> mat = rand(5, 4)
5×4 Matrix{Float64}:
 0.8386    0.83612   0.0353994  0.15547
 0.590172  0.611815  0.0691152  0.915788
 0.879395  0.07271   0.980079   0.655158
 0.340435  0.756196  0.0697535  0.388578
 0.714515  0.861872  0.971521   0.176768

julia> DataFrame(mat, :auto)
5×4 DataFrame
 Row │ x1        x2        x3         x4
     │ Float64   Float64   Float64    Float64
─────┼─────────────────────────────────────────
   1 │ 0.8386    0.83612   0.0353994  0.15547
   2 │ 0.590172  0.611815  0.0691152  0.915788
   3 │ 0.879395  0.07271   0.980079   0.655158
   4 │ 0.340435  0.756196  0.0697535  0.388578
   5 │ 0.714515  0.861872  0.971521   0.176768

Exercise 2

Now, using matrix mat create a data frame with randomly generated column names. Use the randstring function from the Random module to generate them. Store this data frame in df variable.

Solution

julia> using Random

julia> df = DataFrame(mat, [randstring() for _ in 1:4])
5×4 DataFrame
 Row │ 6mTK5evn  K8Inf7ER  5Caz55k0   SRiGemsa
     │ Float64   Float64   Float64    Float64
─────┼─────────────────────────────────────────
   1 │ 0.8386    0.83612   0.0353994  0.15547
   2 │ 0.590172  0.611815  0.0691152  0.915788
   3 │ 0.879395  0.07271   0.980079   0.655158
   4 │ 0.340435  0.756196  0.0697535  0.388578
   5 │ 0.714515  0.861872  0.971521   0.176768

Exercise 3

Create a new data frame, taking df as a source that will have the same columns but its column names will be y1, y2, y3, y4.

Solution

julia> DataFrame(["y$i" => df[!, i] for i in 1:4])
5×4 DataFrame
 Row │ y1        y2        y3         y4
     │ Float64   Float64   Float64    Float64
─────┼─────────────────────────────────────────
   1 │ 0.8386    0.83612   0.0353994  0.15547
   2 │ 0.590172  0.611815  0.0691152  0.915788
   3 │ 0.879395  0.07271   0.980079   0.655158
   4 │ 0.340435  0.756196  0.0697535  0.388578
   5 │ 0.714515  0.861872  0.971521   0.176768

You could also use the raname function:

julia> rename(df, string.("y", 1:4))
5×4 DataFrame
 Row │ y1        y2        y3         y4
     │ Float64   Float64   Float64    Float64
─────┼─────────────────────────────────────────
   1 │ 0.8386    0.83612   0.0353994  0.15547
   2 │ 0.590172  0.611815  0.0691152  0.915788
   3 │ 0.879395  0.07271   0.980079   0.655158
   4 │ 0.340435  0.756196  0.0697535  0.388578
   5 │ 0.714515  0.861872  0.971521   0.176768

Exercise 4

Create a dictionary holding column_name => column_vector pairs using data stored in data frame df. Save this dictionary in variable d.

Solution

julia> d = Dict([n => df[:, n] for n in names(df)])
Dict{String, Vector{Float64}} with 4 entries:
  "6mTK5evn" => [0.8386, 0.590172, 0.879395, 0.340435, 0.714515]
  "5Caz55k0" => [0.0353994, 0.0691152, 0.980079, 0.0697535, 0.971521]
  "K8Inf7ER" => [0.83612, 0.611815, 0.07271, 0.756196, 0.861872]
  "SRiGemsa" => [0.15547, 0.915788, 0.655158, 0.388578, 0.176768]

or (using the pairs function; note that this time column names are Symbol):

julia> Dict(pairs(eachcol(df)))
Dict{Symbol, AbstractVector} with 4 entries:
  Symbol("6mTK5evn") => [0.8386, 0.590172, 0.879395, 0.340435, 0.714515]
  :SRiGemsa          => [0.15547, 0.915788, 0.655158, 0.388578, 0.176768]
  :K8Inf7ER          => [0.83612, 0.611815, 0.07271, 0.756196, 0.861872]
  Symbol("5Caz55k0") => [0.0353994, 0.0691152, 0.980079, 0.0697535, 0.971521]

Exercise 5

Create a data frame back from dictionary d from exercise 4. Compare it with df.

Solution

julia> DataFrame(d)
5×4 DataFrame
 Row │ 5Caz55k0   6mTK5evn  K8Inf7ER  SRiGemsa
     │ Float64    Float64   Float64   Float64
─────┼─────────────────────────────────────────
   1 │ 0.0353994  0.8386    0.83612   0.15547
   2 │ 0.0691152  0.590172  0.611815  0.915788
   3 │ 0.980079   0.879395  0.07271   0.655158
   4 │ 0.0697535  0.340435  0.756196  0.388578
   5 │ 0.971521   0.714515  0.861872  0.176768

Note that columns of a data frame are now sorted by their names. This is done for Dict objects because such dictionaries do not have a defined order of keys.

Exercise 6

For data frame df compute the dot product between all pairs of its columns. Use the dot function from the LinearAlgebra module.

Solution

julia> using LinearAlgebra

julia> using StatsBase

julia> pairwise(dot, eachcol(df))
4×4 Matrix{Float64}:
 2.45132  1.99944  1.65026   1.50558
 1.99944  2.39336  1.03322   1.18411
 1.65026  1.03322  1.9153    0.909744
 1.50558  1.18411  0.909744  1.47431

Exercise 7

Given two data frames:

julia> df1 = DataFrame(a=1:2, b=11:12)
2×2 DataFrame
 Row │ a      b
     │ Int64  Int64
─────┼──────────────
   1 │     1     11
   2 │     2     12

julia> df2 = DataFrame(a=1:2, c=101:102)
2×2 DataFrame
 Row │ a      c
     │ Int64  Int64
─────┼──────────────
   1 │     1    101
   2 │     2    102

vertically concatenate them so that only columns that are present in both data frames are kept. Check the documentation of vcat to see how to do it.

Solution

julia> vcat(df1, df2, cols=:intersect)
4×1 DataFrame
 Row │ a
     │ Int64
─────┼───────
   1 │     1
   2 │     2
   3 │     1
   4 │     2

By default you will get an error:

julia> vcat(df1, df2)
ERROR: ArgumentError: column(s) c are missing from argument(s) 1, and column(s) b are missing from argument(s) 2

Exercise 8

Now append to df1 table df2, but add only the columns from df2 that are present in df1. Check the documentation of append! to see how to do it.

Solution

julia> append!(df1, df2, cols=:subset)
4×2 DataFrame
 Row │ a      b
     │ Int64  Int64?
─────┼────────────────
   1 │     1       11
   2 │     2       12
   3 │     1  missing
   4 │     2  missing

Exercise 9

Create a circle data frame, using the push! function that will store 1000 samples of the following process: * draw x and y uniformly and independently from the [-1,1[ interval; * compute a binary variable inside that is true if x^2+y^2 < 1 and is false otherwise.

Compute summary statistics of this data frame.

Solution

circle=DataFrame()
for _ in 1:1000
    x, y = 2rand()-1, 2rand()-1
    inside = x^2 + y^2 < 1
    push!(circle, (x=x, y=y, inside=inside))
end
describe(circle)

We note that mean of variable inside is approximately π.

Exercise 10

Create a scatterplot of circle data frame where its x and y axis will be the plotted points and inside variable will determine the color of the plotted point.

Solution

using Plots
scatter(circle.x, circle.y, color=[i ? "black" : "red" for i in circle.inside], xlabel="x", ylabel="y", legend=false, size=(400, 400))
scatter(circle.x, circle.y, color=[i ? "black" : "red" for i in circle.inside], xlabel="x", ylabel="y", legend=false, aspect_ratio=:equal)

In the solution two ways to plot ensuring the ratio between x and y axis is 1 are shown. Note the differences in the produced output between the two methods.