summerschool_simtech_2023/material/2_tue/testing/slides.qmd

---
format: revealjs

---

# Learning Goals

- Justify the effort of developing tests to some extent
- Get to know a few common terms of testing
- Work with the Julia unit testing package `Test.jl`

Material is taken and modified, on the one hand, from the [SSE lecture](https://github.com/Simulation-Software-Engineering/Lecture-Material), which builds partly on the [py-rse book](https://merely-useful.tech/py-rse), and, on the other hand, from the [Test.jl docs](https://docs.julialang.org/en/v1/stdlib/Test/).

---

# 1. General Introduction to Testing

---

## What is Testing?

- Smelling old milk before using it!
- A way to determine if a software is not producing reliable results and if so, what is the reason.
- Manual testing vs. automated testing.

---

## Why Should you Test your Software?

- Improve software reliability and reproducibility.
- Make sure that changes (bugfixes, new features) do not affect other parts of software.
- Generally all software is better off being tested regularly. Possible exceptions are very small codes with single users.
- Ensure that a released version of a software actually works.

---

## Nomenclature in Software Testing

- **Fixture**: preparatory set for testing.
- **Actual result**: what the code produces when given the fixture.
- **Expected result**: what the actual result is compared to.
- **Test coverage**: how much of the code do tests touch in one run.

---

## Some Ways to Test Software

- Assertions
- Unit testing
- Integration testing
- Regression testing

---

## Assertions

- Principle of *defensive programming*.
- Nothing happens when an assertion is true; throws error when false.
- Types of assertion statements:
    - Precondition
    - Postcondition
    - Invariant
- A basic but powerful tool to test a software on-the-go.
- Assertion statement syntax in Python

```julia
@assert condition "message"
```

---

## Unit Testing

- Catching errors with assertions is good but preventing them is better!
- A *unit* is a single function in one situation.
    - A situation is one amongst many possible variations of input parameters.
- User creates the expected result manually.
- Fixture is the set of inputs used to generate an actual result.
- Actual result is compared to the expected result by `@test`.

---

## Integration Testing

- Test whether several units work in conjunction.
- *Integrate* units and test them together in an *integration* test.
- Often more complicated than a unit test and has more test coverage.
- A fixture is used to generate an actual result.
- Actual result is compared to the expected result by `@test`.

---

## Regression Testing

- Generating an expected result is not possible in some situations.
- Compare the current actual result with a previous actual result.
- No guarantee that the current actual result is correct.
- Risk of a bug being carried over indefinitely.
- Main purpose is to identify changes in the current state of the code with respect to a past state.

---

## Test Coverage

- Coverage is the amount of code a test touches in one run.
- Aim for high test coverage.
- There is a trade-off: high test coverage vs. effort in test development

---

## Comparing Floating-point Variables

- Very often quantities in math software are `float` / `double`.
- Such quantities cannot be compared to exact values, an approximation is necessary.
- Comparison of floating point variables needs to be done to a certain tolerance.

```julia
@test 1 ≈ 0.999999  rtol=1e-5
```

- Get `≈` by Latex `\approx` + TAB

---

## Test-driven Development (TDD)

- Principle is to write a test and then write a code to fulfill the test.
- Advantages:
    - In the end user ends up with a test alongside the code.
    - Eliminates confirmation bias of the user.
    - Writing tests gives clarity on what the code is supposed to do.
- Disadvantage: known to not improve productivity.

---

## Checking-driven Development (CDD)

- Developer performs spot checks; sanity checks at intermediate stages
- Math software often has heuristics which are easy to determine.
- Keep performing same checks at different stages of development to ensure the code works.

---

## Verifying a Test

- Test written as part of a bug-fix:
    - Reproduce the bug in the test by ensuring that the test fails.
    - Fix the bug.
    - Rerun the test to ensure that it passes.
- Test written to increase code coverage:
    - Make sure that the first iteration of the test passes.
    - Try introducing a small fixable bug in the code to verify if the test fails.

---

# 2. Unit Testing in Julia with Test.jl

---

## Setup of Tests.jl

- Standard library to write and manage tests, `using Test`
- Standardized folder structure:

```
├── Manifest.toml
├── Project.toml
├── src/
└── test
    ├── Manifest.toml
    ├── Project.toml
    ├── runtests.jl
    └── setup.jl
```

- Singular `test` vs plural `runtests.jl`
- `setup.jl` for all `using XYZ` statements, included in `runtests.jl`
- Additional packages either in `[extra] section` of `./Project.toml` or in a new `./test/Project.toml` environment
  - In case of the latter: Do not add the package itself to the `./test/Project.toml`


---

## Run Tests

Various options:

- Directly call `runtests.jl` TODO?
- From Pkg-Manager `]test` when root project is activated

---

## Implement and Structure Tests

- `@test expr`: Test whether expression `expr` is true
- `@test expr broken=true`: Explicitly mark test as broken
- `@test_throws exception expr`: Test whether expression `expr` throws `exception` (test unhappy path)

    ```julia
    julia> @test_throws DimensionMismatch [1, 2, 3] + [1, 2]
    Test Passed
          Thrown: DimensionMismatch
    ```

- `@testset`: Structure tests

   ```julia
   julia> @testset "trigonometric identities" begin
           θ = 2/3*π
           @test sin(-θ) ≈ -sin(θ)
           @test cos(-θ) ≈ cos(θ)
       end;
   ```

- `@testset for ... end`: Test in loop

---

## Further Reading and Watching

- [Research Software Engineering with Python - Chapter 11: Testing Software](https://merely-useful.tech/py-rse/testing.html)
- [HiRSE-Summer of Testing Part 2b: "Testing with Julia" by Nils Niggemann](https://www.youtube.com/watch?v=gSMKNbZOpZU)
- [Official documentation of Test.jl](https://docs.julialang.org/en/v1/stdlib/Test/)

# 3. Test.jl Demo

We use [`MyTestPackage`](https://github.com/s-ccs/summerschool_simtech_2023/tree/main/material/2_tue/testing/MyTestPackage), which looks as follows:


```
├── Manifest.toml
├── Project.toml
├── src
│   ├── find.jl
│   └── MyTestPackage.jl
└── test
    ├── find.jl
    ├── Manifest.toml
    ├── Project.toml
    ├── runtests.jl
    └── setup.jl
```


- Look at `MyTestPackage.jl` and `find.jl`: We have two functions `find_max` and `find_mean`, which calculate the maximum and mean of all elements of a `::AbstractVector`.
  - Assertions were added to check for `NaN` values
- Look at `runtests.jl`:
  - TODO: Why do we need `using MyTestPackage`?
  - We include dependencies via `setup.jl`: `Test` and `StableRNG`.
  - Testset "find"
- Look at `find.jl`
  - Unit tests for `find_max` and `find_mean`
  - `test_throws` to test unhappy path
  - Test with absolute tolerance
  - Integration test, which tests combination of both methods
- Run tests:

    ```
    ]activate .
    ]test
    ```

---

# 4. Exercise

Write tests for your own statistics package 😊