Merge branch 'main' of https://github.com/s-ccs/summerschool_simtech_2023
This commit is contained in:
behinger (s-ccs 001) committed
commit
ec20c173aa
9 files changed
+535
-18
No files matched your search
+2
-2
@@ -57,9 +57,9 @@ website:
|
||||
text: "📝 1 - Advanced Git and Contributing"
|
||||
- href: "material/2_tue/git/tasks.qmd"
|
||||
text: "🛠 1 - Git: Exercises"
|
||||
- href: "material/2_tue/testing/slides.qmd"
|
||||
- href: "material/2_tue/testing/slides.md"
|
||||
text: "📝 2 - Testing"
|
||||
- href: "material/2_tue/CI/missing.qmd"
|
||||
- href: "material/2_tue/ci/slides.md"
|
||||
text: "📝 3 - Continuous Integration"
|
||||
- href: material/2_tue/codereview/slides.qmd
|
||||
text: "📝 4 - Code Review"
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
There are many good ones out there. One we can recommend is the [one from GitHub](https://education.github.com/git-cheat-sheet-education.pdf).
|
||||
@@ -0,0 +1 @@
|
||||
Also [one from GitHub](https://github.github.io/actions-cheat-sheet/actions-cheat-sheet.pdf)
|
||||
@@ -0,0 +1,395 @@
|
||||
---
|
||||
type: slide
|
||||
slideOptions:
|
||||
transition: slide
|
||||
width: 1400
|
||||
height: 900
|
||||
margin: 0.1
|
||||
---
|
||||
|
||||
<style>
|
||||
.reveal strong {
|
||||
font-weight: bold;
|
||||
color: orange;
|
||||
}
|
||||
.reveal p {
|
||||
text-align: left;
|
||||
}
|
||||
.reveal section h1 {
|
||||
color: orange;
|
||||
}
|
||||
.reveal section h2 {
|
||||
color: orange;
|
||||
}
|
||||
.reveal code {
|
||||
font-family: 'Ubuntu Mono';
|
||||
color: orange;
|
||||
}
|
||||
.reveal section img {
|
||||
background:none;
|
||||
border:none;
|
||||
box-shadow:none;
|
||||
}
|
||||
</style>
|
||||
|
||||
# Learning Goals
|
||||
|
||||
- Name and explain common workflows to automate in RSE.
|
||||
- Explain the differences between the various continuous methodologies.
|
||||
- Explain why automation is crucial in RSE.
|
||||
- Write and understand basic automation scripts for GitHub Actions.
|
||||
- s.t. we understand what `PkgTemplates` generates for us.
|
||||
|
||||
|
||||
Material is taken and modified from the [SSE lecture](https://github.com/Simulation-Software-Engineering/Lecture-Material).
|
||||
|
||||
---
|
||||
|
||||
# 1. Workflow Automation
|
||||
|
||||
---
|
||||
|
||||
## Why Automation?
|
||||
|
||||
- Automatize tasks
|
||||
- Run tests frequently, give feedback early etc.
|
||||
- Ensure reproducible test environments
|
||||
- Cannot forget automatized tasks
|
||||
- Less burden to developer (and their workstation)
|
||||
- Avoid manual errors
|
||||
- Process often integrated in development workflow
|
||||
- Example: Support by Git hooks or Git forges
|
||||
|
||||
---
|
||||
|
||||
## Typical Automation Tasks in RSE
|
||||
|
||||
- Check code formatting and quality
|
||||
- Compile and test code for different platforms
|
||||
- Generate coverage reports and visualization
|
||||
- Build documentation and deploy it
|
||||
- Build, package, and upload releases
|
||||
|
||||
---
|
||||
|
||||
## Continuous Methodologies (1/2)
|
||||
|
||||
- **Continuous Integration** (CI)
|
||||
- Continuously integrate changes into "main" branch
|
||||
- Avoids "merge hell"
|
||||
- Relies on testing and checking code continuously
|
||||
- Should be automatized
|
||||
|
||||
---
|
||||
|
||||
## Continuous Methodologies (2/2)
|
||||
|
||||
- **Continuous Delivery** (CD)
|
||||
- Software is in a state that allows new release at any time
|
||||
- Software package is built
|
||||
- Actual release triggered manually
|
||||
- **Continuous Deployment** (CD)
|
||||
- Software is in a state that allows new release at any time
|
||||
- Software package is built
|
||||
- Actual release triggered automatically (continuously)
|
||||
|
||||
---
|
||||
|
||||
## Automation Services/Software
|
||||
|
||||
- [GitHub Actions](https://github.com/features/actions)
|
||||
- [GitLab CI/CD](https://docs.gitlab.com/ee/ci/)
|
||||
- [Circle CI](https://circleci.com/)
|
||||
- [Travis CI](https://www.travis-ci.com/)
|
||||
- [Jenkins](https://www.jenkins.io/)
|
||||
- ...
|
||||
|
||||
---
|
||||
|
||||
# 2. GitHub Actions
|
||||
|
||||
---
|
||||
|
||||
## What is "GitHub Actions"?
|
||||
|
||||
> Automate, customize, and execute your software development workflows right in your repository with GitHub Actions.
|
||||
|
||||
From: [https://docs.github.com/en/actions](https://docs.github.com/en/actions)
|
||||
|
||||
---
|
||||
|
||||
## General Information
|
||||
|
||||
- Usage of GitHub's runners is [limited](https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits)
|
||||
- Available for public repositories or accounts with subscription
|
||||
- By default Actions run on GitHub's runners
|
||||
- Linux, Windows, or MacOS
|
||||
- Quickly evolving and significant improvements in recent years
|
||||
|
||||
---
|
||||
|
||||
## Components (1/2)
|
||||
|
||||
- [Workflow](https://docs.github.com/en/actions/using-workflows): Runs one or more jobs
|
||||
- [Event](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows): Triggers a workflow
|
||||
- [Jobs](https://docs.github.com/en/actions/using-jobs): Set of steps (running on same runner)
|
||||
- Steps executed consecutively and share data
|
||||
- Jobs by default executed in parallel
|
||||
- [Action](https://docs.github.com/en/actions/creating-actions): Application performing common, complex task (step) often used in workflows
|
||||
- [Runner](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions#runners): Server that runs jobs
|
||||
- [Artifacts](https://docs.github.com/en/actions/learn-github-actions/essential-features-of-github-actions#sharing-data-between-jobs): Files to be shared between jobs or to be kept after workflow finishes
|
||||
|
||||
---
|
||||
|
||||
## Components (2/2)
|
||||
|
||||
<img src="https://docs.github.com/assets/cb-25535/mw-1440/images/help/actions/overview-actions-simple.webp" width=95%; style="margin-left:auto; margin-right:auto; padding-top: 25px; padding-bottom: 25px; background: #eeeeee">
|
||||
|
||||
|
||||
From [GitHub Actions tutorial](https://docs.github.com/en/actions)
|
||||
|
||||
---
|
||||
|
||||
## Setting up a Workflow
|
||||
|
||||
- Workflow file files stored `${REPO_ROOT}/.github/workflows`
|
||||
- Configured via YAML file
|
||||
|
||||
```yaml
|
||||
name: learn-github-actions
|
||||
on: [push]
|
||||
jobs:
|
||||
check-bats-version:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
- uses: actions/setup-node@v2
|
||||
with:
|
||||
node-version: '14'
|
||||
- run: npm install -g bats
|
||||
- run: bats -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Actions
|
||||
|
||||
```yaml
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-node@v2
|
||||
with:
|
||||
node-version: '14'
|
||||
```
|
||||
|
||||
- Integrated via `uses` directive
|
||||
- Additional configuration via `with` (options depend on Action)
|
||||
- Find actions in [marketplace](https://github.com/marketplace?type=actions)
|
||||
- Write [own actions](https://docs.github.com/en/actions/creating-actions)
|
||||
|
||||
---
|
||||
|
||||
## Some Useful Julia Actions
|
||||
|
||||
- Find on [gitHub.com/julia-actions](https://github.com/julia-actions/)
|
||||
|
||||
```
|
||||
- uses: julia-actions/setup-julia@v1
|
||||
with:
|
||||
version: '1.9'
|
||||
```
|
||||
|
||||
- More:
|
||||
- `cache`: caches `~/.julia/artifacts/*` and `~/.julia/packages/*` to reduce runtime of CI
|
||||
- `julia-buildpkg`: build package
|
||||
- `julia-runtest`: run tests
|
||||
- `julia-format`: format code
|
||||
|
||||
---
|
||||
|
||||
## User-specified Commands
|
||||
|
||||
```yaml
|
||||
- name: "Single line command"
|
||||
run: echo "Single line command"
|
||||
- name: "Multi line command"
|
||||
run: |
|
||||
echo "First line"
|
||||
echo "Second line. Directory ${PWD}"
|
||||
workdir: tmp/
|
||||
shell: bash
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Events
|
||||
|
||||
- Single or multiple events
|
||||
|
||||
```yaml
|
||||
on: [push, fork]
|
||||
```
|
||||
|
||||
- Activities
|
||||
|
||||
```yaml
|
||||
on:
|
||||
issue:
|
||||
types:
|
||||
- opened
|
||||
- labeled
|
||||
```
|
||||
|
||||
- Filters
|
||||
|
||||
```yaml
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- 'releases/**'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Artifacts
|
||||
|
||||
- Data sharing between jobs and data upload
|
||||
- Uploading artifact
|
||||
|
||||
```yaml
|
||||
- name: "Upload artifact"
|
||||
uses: actions/upload-artifact@v2
|
||||
with:
|
||||
name: my-artifact
|
||||
path: my_file.txt
|
||||
retention-days: 5
|
||||
```
|
||||
|
||||
- Downloading artifact
|
||||
|
||||
```yaml
|
||||
- name: "Download a single artifact"
|
||||
uses: actions/download-artifact@v2
|
||||
with:
|
||||
name: my-artifact
|
||||
```
|
||||
|
||||
**Note**: Drop name to download all artifacts
|
||||
|
||||
---
|
||||
|
||||
## Test Actions Locally
|
||||
|
||||
- [act](https://github.com/nektos/act)
|
||||
- Relies extensively on Docker
|
||||
- User should be in `docker` group
|
||||
- Run `act` from root of the repository
|
||||
|
||||
```text
|
||||
act (runs all workflows)
|
||||
act --job WORKFLOWNAME
|
||||
```
|
||||
|
||||
- Environment is not 100% identical to GitHub's
|
||||
- Workflows may fail locally, but work on GitHub
|
||||
|
||||
---
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [What is Continuous Integration?](https://www.atlassian.com/continuous-delivery/continuous-integration)
|
||||
- [GitHub Actions documentation](https://docs.github.com/en/actions)
|
||||
- [GitHub Actions quickstart](https://docs.github.com/en/actions/quickstart)
|
||||
|
||||
---
|
||||
|
||||
# 3. Demo: Automation with GitHub Actions
|
||||
|
||||
---
|
||||
|
||||
## Setting up a Test Job
|
||||
|
||||
- Import [Julia test package repository](https://github.com/uekerman/JuliaTestPackage) (the same code we used for testing)
|
||||
- Set up workflow file
|
||||
|
||||
```bash
|
||||
mkdir -p .github/workflows
|
||||
cd .github/workflows
|
||||
vi format-check.yml
|
||||
```
|
||||
|
||||
- Let's check whether our code is formatted correctly. Edit `format-check.yml` to have following content
|
||||
|
||||
```yaml
|
||||
name: format-check
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
format:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: julia-actions/setup-julia@v1
|
||||
with:
|
||||
version: '1.9'
|
||||
- name: Install JuliaFormatter and format
|
||||
run: |
|
||||
julia -e 'using Pkg; Pkg.add(PackageSpec(name="JuliaFormatter"))'
|
||||
julia -e 'using JuliaFormatter; format(".", verbose=true)'
|
||||
- name: Format check
|
||||
run: |
|
||||
julia -e '
|
||||
out = Cmd(`git diff --name-only`) |> read |> String
|
||||
if out == ""
|
||||
exit(0)
|
||||
else
|
||||
@error "Some files have not been formatted"
|
||||
write(stdout, out)
|
||||
exit(1)
|
||||
end'
|
||||
```
|
||||
|
||||
- `runs-on` does **not** refer to a Docker container, but to a runner tag.
|
||||
- Add, commit, push
|
||||
- After the push, inspect "Action" panel on GitHub repository
|
||||
- GitHub will schedule a run (yellow dot)
|
||||
- Hooray. We have set up our first action.
|
||||
- Failing test example:
|
||||
- Edit settings on GitHub that one can only merge if all tests pass:
|
||||
- Settings -> Branches -> Branch protection rule
|
||||
- Choose `main` branch
|
||||
- Enable "Require status checks to pass before merging". Optionally enable "Require branches to be up to date before merging"
|
||||
- Choose status checks that need to pass: `test`
|
||||
- Click on "Create" at bottom of page.
|
||||
- Create a new branch `break-code`.
|
||||
- Edit some file, violate the formatting, commit it and push it to the branch. Afterwards open a new PR and inspect the failing test. We are also not able to merge the changes as the "Merge" button should be inactive.
|
||||
|
||||
---
|
||||
|
||||
## act Demo
|
||||
|
||||
- `act` is for quick checks while developing workflows, not for developing the code
|
||||
- Check available jobs (at root of repository)
|
||||
|
||||
```bash
|
||||
act -l
|
||||
```
|
||||
|
||||
- Run jobs for `push` event (default event)
|
||||
|
||||
```bash
|
||||
act
|
||||
```
|
||||
|
||||
- Run a specific job
|
||||
|
||||
```bash
|
||||
act -j test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# 4. Exercise
|
||||
|
||||
Set up GitHub Actions for your statistics package. They should format your code and run the tests. To structure and parallelize things, you could use two separate jobs.
|
||||
@@ -32,7 +32,7 @@ slideOptions:
|
||||
}
|
||||
</style>
|
||||
|
||||
## Learning Goals of the Git Lecture
|
||||
# Learning Goals
|
||||
|
||||
- Refresh and organize students' existing knowledge on Git (learn how to learn more).
|
||||
- Students can explain difference between merge and rebase and when to use what.
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
1. Work with any forge that you like and create a user account (we strongly recommend GitHub since we will need it later again).
|
||||
2. Push your package `MyStatsPackage` to a remote repository.
|
||||
3. Add a function `printOwner` to the package through a pull request. The function should print your (GitHub) user name (hard-coded).
|
||||
4. Use the package from somebody else in the classroom and verify with `printOwner` that you use the correct package.
|
||||
5. Fork this other package and contribute a function `printContributor` to it via a PR. Get a review and get it merged.
|
||||
6. Add more functions to other packages of classmates that print funny things, but always ensure a linear history.
|
||||
4. Start a new Julia environment and use your package through its url: `]add https://github.com/[username]/MyStatsPackage`.
|
||||
5. Now use the package from somebody else in the classroom instead and verify with `printOwner` that you use the correct package.
|
||||
6. Fork this other package and contribute a function `printContributor` to it via a PR. Get a review and get it merged.
|
||||
7. Add more functions to other packages of classmates that print funny things, but always ensure a linear history.
|
||||
@@ -1,9 +1,37 @@
|
||||
|
||||
---
|
||||
format: revealjs
|
||||
|
||||
type: slide
|
||||
slideOptions:
|
||||
transition: slide
|
||||
width: 1400
|
||||
height: 900
|
||||
margin: 0.1
|
||||
---
|
||||
|
||||
<style>
|
||||
.reveal strong {
|
||||
font-weight: bold;
|
||||
color: orange;
|
||||
}
|
||||
.reveal p {
|
||||
text-align: left;
|
||||
}
|
||||
.reveal section h1 {
|
||||
color: orange;
|
||||
}
|
||||
.reveal section h2 {
|
||||
color: orange;
|
||||
}
|
||||
.reveal code {
|
||||
font-family: 'Ubuntu Mono';
|
||||
color: orange;
|
||||
}
|
||||
.reveal section img {
|
||||
background:none;
|
||||
border:none;
|
||||
box-shadow:none;
|
||||
}
|
||||
</style>
|
||||
|
||||
# Learning Goals
|
||||
|
||||
- Justify the effort of developing tests to some extent
|
||||
@@ -0,0 +1,52 @@
|
||||
############################################################################
|
||||
#### Execute code chunks separately in VSCODE by pressing 'Alt + Enter' ####
|
||||
############################################################################
|
||||
|
||||
using Statistics
|
||||
using Plots
|
||||
using RDatasets
|
||||
using GLM
|
||||
|
||||
##
|
||||
|
||||
trees = dataset("datasets", "trees")
|
||||
|
||||
scatter(trees.Girth, trees.Volume,
|
||||
legend=false, xlabel="Girth", ylabel="Volume")
|
||||
|
||||
##
|
||||
|
||||
scatter(trees.Girth, trees.Volume,
|
||||
legend=false, xlabel="Girth", ylabel="Volume")
|
||||
plot!(x -> -37 + 5*x)
|
||||
|
||||
##
|
||||
|
||||
linmod1 = lm(@formula(Volume ~ Girth), trees)
|
||||
|
||||
##
|
||||
|
||||
linmod2 = lm(@formula(Volume ~ Girth + Height), trees)
|
||||
|
||||
##
|
||||
|
||||
r2(linmod1)
|
||||
r2(linmod2)
|
||||
|
||||
linmod3 = lm(@formula(Volume ~ Girth + Height + Girth*Height), trees)
|
||||
|
||||
r2(linmod3)
|
||||
|
||||
##
|
||||
|
||||
using CSV
|
||||
using HTTP
|
||||
|
||||
http_response = HTTP.get("https://vincentarelbundock.github.io/Rdatasets/csv/AER/SwissLabor.csv")
|
||||
SwissLabor = DataFrame(CSV.File(http_response.body))
|
||||
|
||||
SwissLabor[!,"participation"] .= (SwissLabor.participation .== "yes")
|
||||
|
||||
##
|
||||
|
||||
model = glm(@formula(participation ~ age), SwissLabor, Binomial(), ProbitLink())
|
||||
@@ -10,13 +10,26 @@ editor:
|
||||
|
||||
### Introductory Example: tree dataset from R
|
||||
|
||||
\[figure of raw data\]
|
||||
```{julia}
|
||||
using Statistics
|
||||
using Plots
|
||||
using RDatasets
|
||||
|
||||
trees = dataset("datasets", "trees")
|
||||
|
||||
scatter(trees.Volume, trees.Girth,
|
||||
legend=false, xlabel="Girth", ylabel="Volume")
|
||||
```
|
||||
|
||||
*Aim:* Find relationship between the *response variable* `volume` and
|
||||
the *explanatory variable/covariate* `girth`? Can we predict the volume
|
||||
of a tree given its girth?
|
||||
|
||||
\[figure including a straight line\]
|
||||
```{julia}
|
||||
scatter(trees.Girth, trees.Volume,
|
||||
legend=false, xlabel="Girth", ylabel="Volume")
|
||||
plot!(x -> -37 + 5*x)
|
||||
```
|
||||
|
||||
First Guess: There is a linear relation!
|
||||
|
||||
@@ -55,6 +68,10 @@ rather use Julia to solve the problem.
|
||||
\[use Julia code (existing package) to perform linear regression for
|
||||
`volume ~ girth`\]
|
||||
|
||||
```{julia}
|
||||
lm(@formula(Volume ~ Girth), trees)
|
||||
```
|
||||
|
||||
*Interpretation of the Julia output:*
|
||||
|
||||
- column `estimate` : least square estimates for $\hat \beta_0$ and
|
||||
@@ -166,6 +183,15 @@ the corresponding standard errors and the $t$-statistics. Test your
|
||||
functions with the \`\`\`tree''' data set and try to reproduce the
|
||||
output above.
|
||||
|
||||
```{julia}
|
||||
r2(linmod1)
|
||||
r2(linmod2)
|
||||
|
||||
linmod3 = lm(@formula(Volume ~ Girth + Height + Girth*Height), trees)
|
||||
|
||||
r2(linmod3)
|
||||
```
|
||||
|
||||
## Generalized Linear Models
|
||||
|
||||
Classical linear model
|
||||
@@ -206,29 +232,31 @@ $$
|
||||
|
||||
For the models above, these are:
|
||||
|
||||
+----------------------+---------------------+----------------------+
|
||||
+--------------+---------------------+--------------------+
|
||||
| Type of Data | Distribution Family | Link Function |
|
||||
+======================+=====================+======================+
|
||||
+==============+=====================+====================+
|
||||
| continuous | Normal | identity: |
|
||||
| | | |
|
||||
| | | $$ |
|
||||
| | | g(x)=x |
|
||||
| | | $$ |
|
||||
+----------------------+---------------------+----------------------+
|
||||
+--------------+---------------------+--------------------+
|
||||
| count | Poisson | log: |
|
||||
| | | |
|
||||
| | | $$ |
|
||||
| | | g(x) = \log(x) |
|
||||
| | | $$ |
|
||||
+----------------------+---------------------+----------------------+
|
||||
+--------------+---------------------+--------------------+
|
||||
| binary | Bernoulli | logit: |
|
||||
| | | |
|
||||
| | | $$ |
|
||||
| | | g(x) = \log\left |
|
||||
| | | ( |
|
||||
| | | \frac{x}{1-x}\right) |
|
||||
| | | \ |
|
||||
| | | f |
|
||||
| | | rac{x}{1-x}\right) |
|
||||
| | | $$ |
|
||||
+----------------------+---------------------+----------------------+
|
||||
+--------------+---------------------+--------------------+
|
||||
|
||||
In general, the parameter vector $\beta$ is estimated via maximizing the
|
||||
likelihood, i.e.,
|
||||
@@ -246,7 +274,18 @@ $$
|
||||
In the Gaussian case, the maximum likelihood estimator is identical to
|
||||
the least squares estimator considered above.
|
||||
|
||||
\[\[ Example in Julia: maybe `SwissLabor` \]\]
|
||||
```{julia}
|
||||
using CSV
|
||||
using HTTP
|
||||
|
||||
http_response = HTTP.get("https://vincentarelbundock.github.io/Rdatasets/csv/AER/SwissLabor.csv")
|
||||
SwissLabor = DataFrame(CSV.File(http_response.body))
|
||||
|
||||
SwissLabor[!,"participation"] .= (SwissLabor.participation .== "yes")
|
||||
|
||||
model = glm(@formula(participation ~ age^2),
|
||||
SwissLabor, Binomial(), ProbitLink())
|
||||
```
|
||||
|
||||
**Task 3:** Reproduce the results of our data analysis of the `tree`
|
||||
data set using a generalized linear model with normal distribution
|
||||
|
||||
Reference in new issue
Block a user