daviddoji/summerschool_simtech_2023

Files

Benjamin Uekermann 40f1eaca1c Revert rendering as slides

2023-10-09 09:03:54 +02:00

7.8 KiB

Raw Blame History

Learning Goals

Justify the effort of developing tests to some extent
Get to know a few common terms of testing
Work with the Julia unit testing package Test.jl

Material is taken and modified, on the one hand, from the SSE lecture, which builds partly on the py-rse book, and, on the other hand, from the Test.jl docs.

1. General Introduction to Testing

What is Testing?

Smelling old milk before using it!
A way to determine if a software is not producing reliable results and if so, what is the reason.
Manual testing vs. automated testing.

Why Should you Test your Software?

Improve software reliability and reproducibility.
Make sure that changes (bugfixes, new features) do not affect other parts of software.
Generally all software is better off being tested regularly. Possible exceptions are very small codes with single users.
Ensure that a released version of a software actually works.

Nomenclature in Software Testing

Fixture: preparatory set for testing.
Actual result: what the code produces when given the fixture.
Expected result: what the actual result is compared to.
Test coverage: how much of the code do tests touch in one run.

Some Ways to Test Software

Assertions
Unit testing
Integration testing
Regression testing

Assertions

Principle of defensive programming.
Nothing happens when an assertion is true; throws error when false.
Types of assertion statements:
- Precondition
- Postcondition
- Invariant
A basic but powerful tool to test a software on-the-go.
Assertion statement syntax in Python

@assert condition "message"

Unit Testing

Catching errors with assertions is good but preventing them is better!
A unit is a single function in one situation.
- A situation is one amongst many possible variations of input parameters.
User creates the expected result manually.
Fixture is the set of inputs used to generate an actual result.
Actual result is compared to the expected result by @test.

Integration Testing

Test whether several units work in conjunction.
Integrate units and test them together in an integration test.
Often more complicated than a unit test and has more test coverage.
A fixture is used to generate an actual result.
Actual result is compared to the expected result by @test.

Regression Testing

Generating an expected result is not possible in some situations.
Compare the current actual result with a previous actual result.
No guarantee that the current actual result is correct.
Risk of a bug being carried over indefinitely.
Main purpose is to identify changes in the current state of the code with respect to a past state.

Test Coverage

Coverage is the amount of code a test touches in one run.
Aim for high test coverage.
There is a trade-off: high test coverage vs. effort in test development

Comparing Floating-point Variables

Very often quantities in math software are float / double.
Such quantities cannot be compared to exact values, an approximation is necessary.
Comparison of floating point variables needs to be done to a certain tolerance.

@test 1 ≈ 0.999999  rtol=1e-5

Get ≈ by Latex \approx + TAB

Test-driven Development (TDD)

Principle is to write a test and then write a code to fulfill the test.
Advantages:
- In the end user ends up with a test alongside the code.
- Eliminates confirmation bias of the user.
- Writing tests gives clarity on what the code is supposed to do.
Disadvantage: known to not improve productivity.

Checking-driven Development (CDD)

Developer performs spot checks; sanity checks at intermediate stages
Math software often has heuristics which are easy to determine.
Keep performing same checks at different stages of development to ensure the code works.

Verifying a Test

Test written as part of a bug-fix:
- Reproduce the bug in the test by ensuring that the test fails.
- Fix the bug.
- Rerun the test to ensure that it passes.
Test written to increase code coverage:
- Make sure that the first iteration of the test passes.
- Try introducing a small fixable bug in the code to verify if the test fails.

2. Unit Testing in Julia with Test.jl

Setup of Tests.jl

Standard library to write and manage tests, using Test
Standardized folder structure:

├── Manifest.toml
├── Project.toml
├── src/
└── test
    ├── Manifest.toml
    ├── Project.toml
    ├── runtests.jl
    └── setup.jl

Singular test vs plural runtests.jl
setup.jl for all using XYZ statements, included in runtests.jl
Additional packages either in [extra] section of ./Project.toml or in a new ./test/Project.toml environment
- In case of the latter: Do not add the package itself to the ./test/Project.toml

Run Tests

Various options:

Directly call runtests.jl TODO?
From Pkg-Manager ]test when root project is activated

Implement and Structure Tests

@test expr: Test whether expression expr is true
@test expr broken=true: Explicitly mark test as broken

@test_throws exception expr: Test whether expression expr throws exception (test unhappy path)

julia> @test_throws DimensionMismatch [1, 2, 3] + [1, 2]
Test Passed
      Thrown: DimensionMismatch

@testset: Structure tests

julia> @testset "trigonometric identities" begin
        θ = 2/3*π
        @test sin(-θ) ≈ -sin(θ)
        @test cos(-θ) ≈ cos(θ)
    end;

@testset for ... end: Test in loop

3. Test.jl Demo

We use MyTestPackage, which looks as follows:

├── Manifest.toml
├── Project.toml
├── src
│   ├── find.jl
│   └── MyTestPackage.jl
└── test
    ├── find.jl
    ├── Manifest.toml
    ├── Project.toml
    ├── runtests.jl
    └── setup.jl

Look at MyTestPackage.jl and find.jl: We have two functions find_max and find_mean, which calculate the maximum and mean of all elements of a ::AbstractVector.
- Assertions were added to check for NaN values
Look at runtests.jl:
- TODO: Why do we need using MyTestPackage?
- We include dependencies via setup.jl: Test and StableRNG.
- Testset “find”
Look at find.jl
- Unit tests for find_max and find_mean
- test_throws to test unhappy path
- Test with absolute tolerance
- Integration test, which tests combination of both methods
Run tests:
```
]activate .
]test
```

4. Exercise

Write tests for your own statistics package 😊

7.8 KiB Raw Blame History