--- --- # Environments ### What is a environment - A list of "installed" packages^[libraries, dlls, .so etc.] at certain versions - sometimes includes the operation system ### **Reproducible** vs. **Replicable** - **Reproducible**: Someone else can get the same results given your code + data - **Replicable**: Someone else can repeat the whole study on new data ### Why do I need it? - Version control - managing multiple projects - testing out things - your Toolbox needs other packages than e.g. your test-environemnt - Reproducibility - other researchers want to know what packages to install - Collaboration - you want to work on the same data / codebase ## Environments in Julia Every folder with an `Project.toml` file has it's own environment (see below) The "base" environment is active by default: ``` julia (@v1.9) pkg> ``` Keep this as empty+tidy as possible! Tip: (you could also start julia by `julia --project="."`) ### Typical commands ##### `activate` Use `activate .` or `activate ./path/to` creates a new `Project.toml` in the selected folder (`.` means current folder), or activates it, if it already exists. ##### `status` (`st`) Shows the currently installed packages ##### `add` Multiple ways to add packages to the `Project.toml`: - `add UnicodePlots` - `add https://github.com/JuliaPlots/UnicodePlots.jl` - specify branch: `add UnicodePlots#unicodeplots-docs` - specify version `add UnicodePlots@3.3` - `add ./path/to/localPackage` ::: callout-note Folders have to be git-repositories, see below. Probably better use `develop` ::: ##### `remove` (`rm`) remove package from `Project.toml` (not from `~/.julia`, use `gc` - garbage collect for this) ##### `develop` - `dev --local UnicodePlots` - `dev ./Path/To/LocalPackage/` ::: callout-note You can't select a branch with `dev` and need to do it manually ::: ::: callout-note You are asking for the difference of `dev ./Path/Package` and `add ./Path/Package`? Good question! `dev` will always track the actual content of the folder - whereas `add` will make a "snapshot" of the last commit in that folder (has to be an git for `add`!). And you have to use `]up` to actually update to new changes ::: ##### `pin` / `free` You can pin versions of packages, so that they are not updated. Unpin with `free` - also undo `develop` by using `free` ##### `instantiate` / `resolve` `instantiate` setup all dependencies in the given `Project.toml`+`Manifest.toml` `resolve` update the `Manifest.toml` to respect the local setup # Project vs. Package | | Project | Package | | --- | --- | ---- | | installable/reuse? | :white_check_mark: | :negative_squared_cross_mark: | | should be reproducible | :white_check_mark: | :negative_squared_cross_mark: | | produces something? | :white_check_mark: | :negative_squared_cross_mark: | | compatabilities declared? | :negative_squared_cross_mark: | :white_check_mark: | | formal requirements in julia? | :negative_squared_cross_mark: | :white_check_mark: | ## `Project.toml` & `Manifest.toml` #### πŸ“„Project.toml The "big picture": keeps track of user-added dependencies (+ compatabilities + header) ``` [deps] PythonCall = "6099a3de-0909-46bc-b1f4-468b9a2dfc0d" RCall = "6f49c342-dc21-5d91-9882-a32aef131414" ``` #### πŸ“„Manifest.toml The "details": keeps track of all versions of all dependencies, and dependencies of dependencies ``` julia_version = "1.9.2" [[deps.AbstractPlutoDingetjes]] deps = ["Pkg"] git-tree-sha1 = "8eaf9f1b4921132a4cff3f36a1d9ba923b14a481" uuid = "6e696c72-6542-2067-7265-42206c756150" version = "1.1.4" [[deps.ArgTools]] uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f" version = "1.1.1" [[deps.BibInternal]] git-tree-sha1 = "3a760b38ba8da19e64d29244f06104823ff26f25" uuid = "2027ae74-3657-4b95-ae00-e2f7d55c3e64" version = "0.3.4" [...] ``` ## Projects in Julia Formally, projects don't have specific requirements. You should activate an environment (`Project.toml`+`Manifest.toml`) in the main folder though. I recommend the following minimal structure: - `./src/` - all functions should go there - `./scripts/` - all actual scripts should go here, - `./README.md` - Write what this is about, who you are etc. - `./Project.toml` - Your explicit dependencies - `./Manifest.toml` - Your implicit dependencies + versions <-- this makes it reproducible! ::: callout-tip One recommendation is to use `DrWatson.initialize_project([path])` to start a new project - it will generate a nice folder structure + provide some other helpful `DrWatson.jl` features. ::: :::{.callout-tip collapse="true"} ## Click to expand the full datastructure ``` β”‚projectdir <- Project's main folder. It is initialized as a Git β”‚ repository with a reasonable .gitignore file. β”‚ β”œβ”€β”€ _research <- WIP scripts, code, notes, comments, β”‚ | to-dos and anything in an alpha state. β”‚ └── tmp <- Temporary data folder. β”‚ β”œβ”€β”€ data <- **Immutable and add-only!** β”‚ β”œβ”€β”€ sims <- Data resulting directly from simulations. β”‚ β”œβ”€β”€ exp_pro <- Data from processing experiments. β”‚ └── exp_raw <- Raw experimental data. β”‚ β”œβ”€β”€ plots <- Self-explanatory. β”œβ”€β”€ notebooks <- Jupyter, Weave or any other mixed media notebooks. β”‚ β”œβ”€β”€ papers <- Scientific papers resulting from the project. β”‚ β”œβ”€β”€ scripts <- Various scripts, e.g. simulations, plotting, analysis, β”‚ β”‚ The scripts use the `src` folder for their base code. β”‚ └── intro.jl <- Simple file that uses DrWatson and uses its greeting. β”‚ β”œβ”€β”€ src <- Source code for use in this project. Contains functions, β”‚ structures and modules that are used throughout β”‚ the project and in multiple scripts. β”‚ β”œβ”€β”€ README.md <- Optional top-level README for anyone using this project. β”œβ”€β”€ .gitignore <- by default ignores _research, data, plots, videos, β”‚ notebooks and latex-compilation related files. β”‚ β”œβ”€β”€ Manifest.toml <- Contains full list of exact package versions used currently. └── Project.toml <- Main project file, allows activation and installation. Includes DrWatson by default. ``` ::: ## Packages in Julia Several thousand packages exist in Julia already. Take a thorough look before starting something new! ### Minimal requirements for `]add` to work Minimal structure One git-repository containing: - `./src/MyStatsPackage.jl` - (`module MyStatsPackage`) - `./Project.toml` - `name = "MyStatsPackage"` - `uuid ="b4cd1eb8-1e24-11e8-3319-93036a3eb9f3"` - (`[compat]` entries) - (`version= "0.1.0"`) ### Additional requirements to register Julia supports many registries (you can host your own!), which are just fancy GITs that index what version is available at what git-url for each registered package. The default registry is [JuliaRegistries/General](https://github.com/JuliaRegistries/General). [To register at the general registiry, you need additionally:](https://juliaregistries.github.io/RegistryCI.jl/stable/guidelines/): - `[compat]` entries for all dependencies - a `version=` - a supported license - Some restrictions on the name (e.g. nothing with `Julia`, only ASCII, etc.) ## Let's generate our first package! ```julia ] generate MyStatsPackage ``` ### Adding dependencies ```julia ]activate ./path/to/MyStatsPackage ]add ProgressMeter ]compat # <1> ``` 1. let's directly add a compat entry for ProgressMeter ### Semantic Versioning Following `semver` - three parts: `v2.7.5` means: - **Major** 2 - **Minor** 7 - **Bugfix** 5 - Bump **Major** if you propose backward-breaking changes - Bump **Minor** if you only introduce new features -Bump **Bugfix** if you, well, fix bugs **Special case:** `v0.37.1` Means package is in development and not stable. Bump **Major** if you release it Bump **Minor** for breaking changes Bump **Bugfix** if you fix bugs or release new features ### Compat entries compat entries define with what versions your package is compatible with ```julia [compat] AllMinorReleases1 = "1" # <1> AllMinorReleases2 = "1.5" #<2> AllMinorReleases3 = "1.5.3" #<3> ExactPackage = "=1.5.6" #<4> MultiVersionexample = "0.5,1.2,2" DevelopPackage = "0.2.3" # <5> ``` 1. [1.0.0-2) 2. [1.5.0-2) 3. [1.5.3-2) 4. [1.5.6] 5. [0.2.3 - 0.3) As you can see, develop version (`version < 1`) are treated a bit special in Julia, and different to `semver`. [Read more here](https://pkgdocs.julialang.org/v1/compatibility/#compat-pre-1.0) ::: callout-warning keep the compat list in alphabetical order - github-actions might behave very strange else. ::: ### Internals of a package The file `./src/MyStatsPackage.jl` should contain: ```julia module MyStatsPackage using ProgressMeter # <1> include("src/stats_functions.jl") export sum export mean, tstat end ``` 1. We use GLMakie as a simple example as you need it on Wednesday again anyway - it does take a while to install though! Now we are ready to use the package from a different environment ```julia ]dev ./path/to/MyStatsPackage ```