Merge branch 'main' of https://github.com/s-ccs/summerschool_simtech_2023

_quarto.yml

@@ -57,9 +57,9 @@ website:
                 text: "📝 1 - Advanced Git and Contributing"
               - href: "material/2_tue/git/tasks.qmd"
                 text: "🛠 1 - Git: Exercises"
-              - href: "material/2_tue/testing/slides.qmd"
+              - href: "material/2_tue/testing/slides.md"
                 text: "📝 2 - Testing"
-              - href: "material/2_tue/CI/missing.qmd"
+              - href: "material/2_tue/ci/slides.md"
                 text: "📝 3 - Continuous Integration"
               - href: material/2_tue/codereview/slides.qmd
                 text: "📝 4 - Code Review"

cheatsheets/git.qmd

@@ -0,0 +1 @@
+There are many good ones out there. One we can recommend is the [one from GitHub](https://education.github.com/git-cheat-sheet-education.pdf).

cheatsheets/githubactions.qmd

@@ -0,0 +1 @@
+Also [one from GitHub](https://github.github.io/actions-cheat-sheet/actions-cheat-sheet.pdf)

material/2_tue/ci/slides.md

@@ -0,0 +1,395 @@
+---
+type: slide
+slideOptions:
+  transition: slide
+  width: 1400
+  height: 900
+  margin: 0.1
+---
+
+<style>
+  .reveal strong {
+    font-weight: bold;
+    color: orange;
+  }
+  .reveal p {
+    text-align: left;
+  }
+  .reveal section h1 {
+    color: orange;
+  }
+  .reveal section h2 {
+    color: orange;
+  }
+  .reveal code {
+    font-family: 'Ubuntu Mono';
+    color: orange;
+  }
+  .reveal section img {
+    background:none;
+    border:none;
+    box-shadow:none;
+  }
+</style>
+
+# Learning Goals
+
+- Name and explain common workflows to automate in RSE.
+- Explain the differences between the various continuous methodologies.
+- Explain why automation is crucial in RSE.
+- Write and understand basic automation scripts for GitHub Actions.
+  - s.t. we understand what `PkgTemplates` generates for us.
+
+
+Material is taken and modified from the [SSE lecture](https://github.com/Simulation-Software-Engineering/Lecture-Material).
+
+---
+
+# 1. Workflow Automation
+
+---
+
+## Why Automation?
+
+- Automatize tasks
+    - Run tests frequently, give feedback early etc.
+    - Ensure reproducible test environments
+    - Cannot forget automatized tasks
+    - Less burden to developer (and their workstation)
+    - Avoid manual errors
+- Process often integrated in development workflow
+    - Example: Support by Git hooks or Git forges
+
+---
+
+## Typical Automation Tasks in RSE
+
+- Check code formatting and quality
+- Compile and test code for different platforms
+- Generate coverage reports and visualization
+- Build documentation and deploy it
+- Build, package, and upload releases
+
+---
+
+## Continuous Methodologies (1/2)
+
+- **Continuous Integration** (CI)
+    - Continuously integrate changes into "main" branch
+    - Avoids "merge hell"
+    - Relies on testing and checking code continuously
+        - Should be automatized
+
+---
+
+## Continuous Methodologies (2/2)
+
+- **Continuous Delivery** (CD)
+    - Software is in a state that allows new release at any time
+    - Software package is built
+    - Actual release triggered manually
+- **Continuous Deployment** (CD)
+    - Software is in a state that allows new release at any time
+    - Software package is built
+    - Actual release triggered automatically (continuously)
+
+---
+
+## Automation Services/Software
+
+- [GitHub Actions](https://github.com/features/actions)
+- [GitLab CI/CD](https://docs.gitlab.com/ee/ci/)
+- [Circle CI](https://circleci.com/)
+- [Travis CI](https://www.travis-ci.com/)
+- [Jenkins](https://www.jenkins.io/)
+- ...
+
+---
+
+# 2. GitHub Actions
+
+---
+
+## What is "GitHub Actions"?
+
+> Automate, customize, and execute your software development workflows right in your repository with GitHub Actions.
+
+From: [https://docs.github.com/en/actions](https://docs.github.com/en/actions)
+
+---
+
+## General Information
+
+- Usage of GitHub's runners is [limited](https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits)
+- Available for public repositories or accounts with subscription
+- By default Actions run on GitHub's runners
+    - Linux, Windows, or MacOS
+- Quickly evolving and significant improvements in recent years
+
+---
+
+## Components (1/2)
+
+- [Workflow](https://docs.github.com/en/actions/using-workflows): Runs one or more jobs
+- [Event](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows): Triggers a workflow
+- [Jobs](https://docs.github.com/en/actions/using-jobs): Set of steps (running on same runner)
+    - Steps executed consecutively and share data
+    - Jobs by default executed in parallel
+- [Action](https://docs.github.com/en/actions/creating-actions): Application performing common, complex task (step) often used in workflows
+- [Runner](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions#runners): Server that runs jobs
+- [Artifacts](https://docs.github.com/en/actions/learn-github-actions/essential-features-of-github-actions#sharing-data-between-jobs): Files to be shared between jobs or to be kept after workflow finishes
+
+---
+
+## Components (2/2)
+
+<img src="https://docs.github.com/assets/cb-25535/mw-1440/images/help/actions/overview-actions-simple.webp" width=95%; style="margin-left:auto; margin-right:auto; padding-top: 25px; padding-bottom: 25px; background: #eeeeee">
+
+
+From [GitHub Actions tutorial](https://docs.github.com/en/actions)
+
+---
+
+## Setting up a Workflow
+
+- Workflow file files stored `${REPO_ROOT}/.github/workflows`
+- Configured via YAML file
+
+```yaml
+name: learn-github-actions
+on: [push]
+jobs:
+  check-bats-version:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-node@v2
+        with:
+          node-version: '14'
+      - run: npm install -g bats
+      - run: bats -v
+```
+
+---
+
+## Actions
+
+```yaml
+- uses: actions/checkout@v3
+- uses: actions/setup-node@v2
+  with:
+    node-version: '14'
+```
+
+- Integrated via `uses` directive
+- Additional configuration via `with` (options depend on Action)
+- Find actions in [marketplace](https://github.com/marketplace?type=actions)
+- Write [own actions](https://docs.github.com/en/actions/creating-actions)
+
+---
+
+## Some Useful Julia Actions
+
+- Find on [gitHub.com/julia-actions](https://github.com/julia-actions/)
+
+   ```
+   - uses: julia-actions/setup-julia@v1
+        with:
+          version: '1.9'
+   ```
+
+- More:
+  - `cache`: caches `~/.julia/artifacts/*` and `~/.julia/packages/*` to reduce runtime of CI
+  - `julia-buildpkg`: build package
+  - `julia-runtest`: run tests
+  - `julia-format`: format code
+
+---
+
+## User-specified Commands
+
+```yaml
+- name: "Single line command"
+  run: echo "Single line command"
+- name: "Multi line command"
+  run: |
+    echo "First line"
+    echo "Second line. Directory ${PWD}"
+  workdir: tmp/
+  shell: bash
+```
+
+---
+
+## Events
+
+- Single or multiple events
+
+  ```yaml
+  on: [push, fork]
+  ```
+
+- Activities
+
+  ```yaml
+  on:
+    issue:
+      types:
+        - opened
+        - labeled
+  ```
+
+- Filters
+
+  ```yaml
+  on:
+    push:
+      branches:
+        - main
+        - 'releases/**'
+  ```
+
+---
+
+## Artifacts
+
+- Data sharing between jobs and data upload
+- Uploading artifact
+
+  ```yaml
+  - name: "Upload artifact"
+    uses: actions/upload-artifact@v2
+    with:
+      name: my-artifact
+      path: my_file.txt
+      retention-days: 5
+  ```
+
+- Downloading artifact
+
+  ```yaml
+  - name: "Download a single artifact"
+    uses: actions/download-artifact@v2
+    with:
+      name: my-artifact
+  ```
+
+  **Note**: Drop name to download all artifacts
+
+---
+
+## Test Actions Locally
+
+- [act](https://github.com/nektos/act)
+- Relies extensively on Docker
+    - User should be in `docker` group
+- Run `act` from root of the repository
+
+  ```text
+  act (runs all workflows)
+  act --job WORKFLOWNAME
+  ```
+
+- Environment is not 100% identical to GitHub's
+    - Workflows may fail locally, but work on GitHub
+
+---
+
+## Further Reading
+
+- [What is Continuous Integration?](https://www.atlassian.com/continuous-delivery/continuous-integration)
+- [GitHub Actions documentation](https://docs.github.com/en/actions)
+- [GitHub Actions quickstart](https://docs.github.com/en/actions/quickstart)
+
+---
+
+# 3. Demo: Automation with GitHub Actions
+
+---
+
+## Setting up a Test Job
+
+- Import [Julia test package repository](https://github.com/uekerman/JuliaTestPackage) (the same code we used for testing)
+- Set up workflow file
+
+  ```bash
+  mkdir -p .github/workflows
+  cd .github/workflows
+  vi format-check.yml
+  ```
+
+- Let's check whether our code is formatted correctly. Edit `format-check.yml` to have following content
+
+  ```yaml
+  name: format-check
+
+  on: [push, pull_request]
+
+  jobs:
+    format:
+      runs-on: ubuntu-latest
+      steps:
+        - uses: actions/checkout@v3
+        - uses: julia-actions/setup-julia@v1
+          with:
+            version: '1.9'
+        - name: Install JuliaFormatter and format
+          run: |
+            julia  -e 'using Pkg; Pkg.add(PackageSpec(name="JuliaFormatter"))'
+            julia  -e 'using JuliaFormatter; format(".", verbose=true)'
+        - name: Format check
+          run: |
+            julia -e '
+            out = Cmd(`git diff --name-only`) |> read |> String
+            if out == ""
+                exit(0)
+            else
+                @error "Some files have not been formatted"
+                write(stdout, out)
+                exit(1)
+            end'     
+  ```
+
+- `runs-on` does **not** refer to a Docker container, but to a runner tag.
+- Add, commit, push
+- After the push, inspect "Action" panel on GitHub repository
+    - GitHub will schedule a run (yellow dot)
+    - Hooray. We have set up our first action.
+- Failing test example:
+    - Edit settings on GitHub that one can only merge if all tests pass:
+        - Settings -> Branches -> Branch protection rule
+        - Choose `main` branch
+        - Enable "Require status checks to pass before merging". Optionally enable "Require branches to be up to date before merging"
+        - Choose status checks that need to pass: `test`
+        - Click on "Create" at bottom of page.
+    - Create a new branch `break-code`.
+    - Edit some file, violate the formatting, commit it and push it to the branch. Afterwards open a new PR and inspect the failing test. We are also not able to merge the changes as the "Merge" button should be inactive.
+
+---
+
+## act Demo
+
+- `act` is for quick checks while developing workflows, not for developing the code
+- Check available jobs (at root of repository) 
+
+  ```bash
+  act -l
+  ```
+
+- Run jobs for `push` event (default event)
+
+  ```bash
+  act
+  ```
+
+- Run a specific job
+
+  ```bash
+  act -j test
+  ```
+
+---
+
+# 4. Exercise
+
+Set up GitHub Actions for your statistics package. They should format your code and run the tests. To structure and parallelize things, you could use two separate jobs.

material/2_tue/git/slides.md

@@ -32,7 +32,7 @@ slideOptions:
   }
 </style>
 
-## Learning Goals of the Git Lecture
+# Learning Goals
 
 - Refresh and organize students' existing knowledge on Git (learn how to learn more).
 - Students can explain difference between merge and rebase and when to use what.

material/2_tue/git/tasks.qmd

@@ -3,6 +3,7 @@
 1. Work with any forge that you like and create a user account (we strongly recommend GitHub since we will need it later again).
 2. Push your package `MyStatsPackage` to a remote repository.
 3. Add a function `printOwner` to the package through a pull request. The function should print your (GitHub) user name (hard-coded).
-4. Use the package from somebody else in the classroom and verify with `printOwner` that you use the correct package.
+4. Start a new Julia environment and use your package through its url: `]add https://github.com/[username]/MyStatsPackage`.
-5. Fork this other package and contribute a function `printContributor` to it via a PR. Get a review and get it merged.
+5. Now use the package from somebody else in the classroom instead and verify with `printOwner` that you use the correct package.
-6. Add more functions to other packages of classmates that print funny things, but always ensure a linear history.
+6. Fork this other package and contribute a function `printContributor` to it via a PR. Get a review and get it merged.
+7. Add more functions to other packages of classmates that print funny things, but always ensure a linear history.

material/2_tue/testing/slides.md

@@ -1,9 +1,37 @@
-
 ---
-format: revealjs
+type: slide
-
+slideOptions:
+  transition: slide
+  width: 1400
+  height: 900
+  margin: 0.1
 ---
 
+<style>
+  .reveal strong {
+    font-weight: bold;
+    color: orange;
+  }
+  .reveal p {
+    text-align: left;
+  }
+  .reveal section h1 {
+    color: orange;
+  }
+  .reveal section h2 {
+    color: orange;
+  }
+  .reveal code {
+    font-family: 'Ubuntu Mono';
+    color: orange;
+  }
+  .reveal section img {
+    background:none;
+    border:none;
+    box-shadow:none;
+  }
+</style>
+
 # Learning Goals
 
 - Justify the effort of developing tests to some extent

material/3_wed/regression/Code_Snippets.jl

@@ -0,0 +1,52 @@
+############################################################################
+#### Execute code chunks separately in VSCODE by pressing 'Alt + Enter' ####
+############################################################################
+
+using Statistics
+using Plots
+using RDatasets
+using GLM
+
+##
+
+trees = dataset("datasets", "trees")
+
+scatter(trees.Girth, trees.Volume,
+        legend=false, xlabel="Girth", ylabel="Volume")
+
+##
+
+scatter(trees.Girth, trees.Volume,
+        legend=false, xlabel="Girth", ylabel="Volume")
+plot!(x -> -37 + 5*x)
+
+##
+
+linmod1 = lm(@formula(Volume ~ Girth), trees)
+
+##
+
+linmod2 = lm(@formula(Volume ~ Girth + Height), trees)
+
+##
+
+r2(linmod1)
+r2(linmod2)
+
+linmod3 = lm(@formula(Volume ~ Girth + Height + Girth*Height), trees)
+
+r2(linmod3)
+
+##
+
+using CSV
+using HTTP
+
+http_response = HTTP.get("https://vincentarelbundock.github.io/Rdatasets/csv/AER/SwissLabor.csv")
+SwissLabor = DataFrame(CSV.File(http_response.body))
+
+SwissLabor[!,"participation"] .= (SwissLabor.participation .== "yes")
+
+##
+
+model = glm(@formula(participation ~ age), SwissLabor, Binomial(), ProbitLink())

material/3_wed/regression/MultipleRegressionBasics.qmd

@@ -10,13 +10,26 @@ editor:
 
 ### Introductory Example: tree dataset from R
 
-\[figure of raw data\]
+```{julia}
+using Statistics
+using Plots
+using RDatasets
+
+trees = dataset("datasets", "trees")
+
+scatter(trees.Volume, trees.Girth,
+        legend=false, xlabel="Girth", ylabel="Volume")
+```
 
 *Aim:* Find relationship between the *response variable* `volume` and
 the *explanatory variable/covariate* `girth`? Can we predict the volume
 of a tree given its girth?
 
-\[figure including a straight line\]
+```{julia}
+scatter(trees.Girth, trees.Volume,
+        legend=false, xlabel="Girth", ylabel="Volume")
+plot!(x -> -37 + 5*x)
+```
 
 First Guess: There is a linear relation!
 
@@ -55,6 +68,10 @@ rather use Julia to solve the problem.
 \[use Julia code (existing package) to perform linear regression for
 `volume ~ girth`\]
 
+```{julia}
+lm(@formula(Volume ~ Girth), trees)
+```
+
 *Interpretation of the Julia output:*
 
 -   column `estimate` : least square estimates for $\hat \beta_0$ and
@@ -166,6 +183,15 @@ the corresponding standard errors and the $t$-statistics. Test your
 functions with the \`\`\`tree''' data set and try to reproduce the
 output above.
 
+```{julia}
+r2(linmod1)
+r2(linmod2)
+
+linmod3 = lm(@formula(Volume ~ Girth + Height + Girth*Height), trees)
+
+r2(linmod3)
+```
+
 ## Generalized Linear Models
 
 Classical linear model
@@ -206,29 +232,31 @@ $$
 
 For the models above, these are:
 
-+----------------------+---------------------+----------------------+
++--------------+---------------------+--------------------+
-| Type of Data         | Distribution Family | Link Function        |
+| Type of Data | Distribution Family | Link Function      |
-+======================+=====================+======================+
++==============+=====================+====================+
-| continuous           | Normal              | identity:            |
+| continuous   | Normal              | identity:          |
-|                      |                     |                      |
+|              |                     |                    |
-|                      |                     | $$                   |
+|              |                     | $$                 |
-|                      |                     | g(x)=x               |
+|              |                     | g(x)=x             |
-|                      |                     | $$                   |
+|              |                     | $$                 |
-+----------------------+---------------------+----------------------+
++--------------+---------------------+--------------------+
-| count                | Poisson             | log:                 |
+| count        | Poisson             | log:               |
-|                      |                     |                      |
+|              |                     |                    |
-|                      |                     | $$                   |
+|              |                     | $$                 |
-|                      |                     |  g(x) = \log(x)      |
+|              |                     |  g(x) = \log(x)    |
-|                      |                     | $$                   |
+|              |                     | $$                 |
-+----------------------+---------------------+----------------------+
++--------------+---------------------+--------------------+
-| binary               | Bernoulli           | logit:               |
+| binary       | Bernoulli           | logit:             |
-|                      |                     |                      |
+|              |                     |                    |
-|                      |                     | $$                   |
+|              |                     | $$                 |
-|                      |                     | g(x) = \log\left     |
+|              |                     | g(x) = \log\left   |
-|                      |                     | (                    |
+|              |                     | (                  |
-|                      |                     | \frac{x}{1-x}\right) |
+|              |                     | \                  |
-|                      |                     | $$                   |
+|              |                     | f                  |
-+----------------------+---------------------+----------------------+
+|              |                     | rac{x}{1-x}\right) |
+|              |                     | $$                 |
++--------------+---------------------+--------------------+
 
 In general, the parameter vector $\beta$ is estimated via maximizing the
 likelihood, i.e.,
@@ -246,7 +274,18 @@ $$
 In the Gaussian case, the maximum likelihood estimator is identical to
 the least squares estimator considered above.
 
-\[\[ Example in Julia: maybe `SwissLabor` \]\]
+```{julia}
+using CSV
+using HTTP
+
+http_response = HTTP.get("https://vincentarelbundock.github.io/Rdatasets/csv/AER/SwissLabor.csv")
+SwissLabor = DataFrame(CSV.File(http_response.body))
+
+SwissLabor[!,"participation"] .= (SwissLabor.participation .== "yes")
+
+model = glm(@formula(participation ~ age^2), 
+            SwissLabor, Binomial(), ProbitLink())
+```
 
 **Task 3:** Reproduce the results of our data analysis of the `tree`
 data set using a generalized linear model with normal distribution