11 lines
34 KiB
JSON
11 lines
34 KiB
JSON
{
|
||
"hash": "4f6a3c2e0c4fb24bd9282d73a347cd19",
|
||
"result": {
|
||
"markdown": "# The SciML suite of packages\n\n\n\nThe `Julia` ecosystem advances rapidly. For much of it, the driving force is the [SciML](https://github.com/SciML) organization (Scientific Machine Learning).\n\n\nIn this section we describe some packages provided by this organization that could be used as alternatives to the ones utilized in these notes. Members of this organization created many packages for solving different types of differential equations, and have branched out from there. Many newer efforts of this organization have been to write uniform interfaces to other packages in the ecosystem, some of which are discussed below. We don't discuss the promise of SCIML: \"Performance is considered a priority, and performance issues are considered bugs,\" as we don't pursue features like in-place modification, sparsity, etc. Interested readers should consult the relevant packages documentation.\n\n\nThe basic structure to use these packages is the \"problem-algorithm-solve\" interface described in [The problem-algorithm-solve interface](../ODEs/solve.html). We also discussed this interface a bit in [ODEs](../ODEs/differential_equations.html).\n\n\n:::{.callout-note}\n## Note\nThese packages are in a process of rapid development and change to them is expected. These notes were written using the following versions:\n\n:::\n\n::: {.cell execution_count=1}\n``` {.julia .cell-code}\npkgs = [\"Symbolics\", \"NonlinearSolve\", \"Optimization\", \"Integrals\"]\nimport Pkg; Pkg.status(pkgs)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n Project CalculusWithJuliaNotes v0.1.3\n Status `~/julia/CalculusWithJuliaNotes/Project.toml` (empty project)\n```\n:::\n:::\n\n\n## Symbolic math (`Symbolics`)\n\n\nThe `Symbolics`, `SymbolicUtils`, and `ModelingToolkit` packages are provided by this organization. These can be viewed as an alternative to `SymPy`, which is used throughout this set of notes. See the section on [Symbolics](./symbolics.html) for some additional details, the package [documentation](https://symbolics.juliasymbolics.org/stable/), or the documentation for [SymbolicsUtils](https://github.com/JuliaSymbolics/SymbolicUtils.jl).\n\n\n## Solving equations\n\n\nSolving one or more equations (simultaneously) is different in the linear case (where solutions are readily found – though performance can distinguish approaches – and the nonlinear case – where for most situations, numeric approaches are required.\n\n\n### `LinearSolve`\n\n\nThe `LinearSolve` package aims to generalize the solving of linear equations. For many cases these are simply represented as matrix equations of the form `Ax=b`, from which `Julia` (borrowing from MATLAB) offers the interface `A \\ b` to yield `x`. There are scenarios that don't naturally fit this structure and perhaps problems where different tolerances need to be specified, and the `LinearSolve` package aims to provide a common interface to handle these scenarios. As this set of notes doesn't bump into such, this package is not described here. In the symbolic case, the `Symbolics.solve_for` function was described in [Symbolics](./symbolics.html).\n\n\n### `NonlinearSolve`\n\n\nThe `NonlinearSolve` package can be seen as an alternative to the use of the `Roots` package in this set of notes. The package presents itself as \"Fast implementations of root finding algorithms in Julia that satisfy the SciML common interface.\"\n\n\nThe package is loaded through the following command:\n\n``` {.julia .cell-code}\nusing NonlinearSolve\n```\n\n\nUnlike `Roots`, the package handles problems beyond the univariate case, as such the simplest problems have a little extra setup required.\n\n\nFor example, suppose we want to use this package to solve for zeros of $f(x) = x^5 - x - 1$. We could do so a few different ways.\n\n\nFirst, we need to define a `Julia` function representing `f`. We do so with:\n\n::: {.cell execution_count=3}\n``` {.julia .cell-code}\nf(u, p) = @. (u^5 - u - 1)\n```\n\n::: {.cell-output .cell-output-display execution_count=4}\n```\nf (generic function with 1 method)\n```\n:::\n:::\n\n\nThe function definition expects a container for the \"`x`\" variables and allows the passing of a container to hold parameters. We could have used the dotted operations for the power and each subtraction to allow vectorization of these basic math operations, as `u` is a container of values. The `@.` macro makes adding the \"dots\" quite easy, as illustrated above. It converts \"every function call or operator in expr into a `dot call`.\"\n\n\nA problem is set up with this function and an initial guess. The `@SVector` specification for the guess is for performance purposes and is provided by the `StaticArrays` package.\n\n::: {.cell execution_count=4}\n``` {.julia .cell-code}\nusing StaticArrays\nu0 = @SVector[1.0]\nprob = NonlinearProblem(f, u0)\n```\n\n::: {.cell-output .cell-output-display execution_count=5}\n\n::: {.ansi-escaped-output}\n```{=html}\n<pre><span class=\"ansi-cyan-fg\">NonlinearProblem</span> with uType <span class=\"ansi-cyan-fg\">SVector{1, Float64}</span>. In-place: <span class=\"ansi-cyan-fg\">false</span>\nu0: 1-element SVector{1, Float64} with indices SOneTo(1):\n 1.0</pre>\n```\n:::\n\n:::\n:::\n\n\nThe problem is solved by calling `solve` with an appropriate method specified. Here we use Newton's method. The derivative of `f` is computed automatically.\n\n::: {.cell execution_count=5}\n``` {.julia .cell-code}\nsoln = solve(prob, NewtonRaphson())\n```\n\n::: {.cell-output .cell-output-display execution_count=6}\n```\nu: 1-element SVector{1, Float64} with indices SOneTo(1):\n 1.1673039782614187\n```\n:::\n:::\n\n\nThe basic interface for retrieving the solution from the solution object is to use indexing:\n\n::: {.cell execution_count=6}\n``` {.julia .cell-code}\nsoln[]\n```\n\n::: {.cell-output .cell-output-display execution_count=7}\n```\n1.1673039782614187\n```\n:::\n:::\n\n\n---\n\n\n:::{.callout-note}\n## Note\nThis interface is more performant than `Roots`, though it isn't an apples to oranges comparison as different stopping criteria are used by the two. In order to be so, we need to help out the call to `NonlinearProblem` to indicate the problem is non-mutating by adding a \"`false`\", as follows:\n\n:::\n\n::: {.cell execution_count=7}\n``` {.julia .cell-code}\nusing BenchmarkTools\n@btime solve(NonlinearProblem{false}(f, @SVector[1.0]), NewtonRaphson())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n 362.966 ns (0 allocations: 0 bytes)\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=8}\n```\nu: 1-element SVector{1, Float64} with indices SOneTo(1):\n 1.1673039782614187\n```\n:::\n:::\n\n\nAs compared to:\n\n::: {.cell execution_count=8}\n``` {.julia .cell-code}\nimport Roots\nimport ForwardDiff\ng(x) = x^5 - x - 1\ngp(x) = ForwardDiff.derivative(g, x)\n@btime solve(Roots.ZeroProblem((g, gp), 1.0), Roots.Newton())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n 411.510 ns (0 allocations: 0 bytes)\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=9}\n```\n1.1673039782614187\n```\n:::\n:::\n\n\n---\n\nThis problem can also be solved using a bracketing method. The package has both `Bisection` and `Falsi` as possible methods. To use a bracketing method, the initial bracket must be specified.\n\n::: {.cell execution_count=9}\n``` {.julia .cell-code}\nu0 = (1.0, 2.0)\nprob = NonlinearProblem(f, u0)\n```\n\n::: {.cell-output .cell-output-display execution_count=10}\n\n::: {.ansi-escaped-output}\n```{=html}\n<pre><span class=\"ansi-cyan-fg\">NonlinearProblem</span> with uType <span class=\"ansi-cyan-fg\">Tuple{Float64, Float64}</span>. In-place: <span class=\"ansi-cyan-fg\">false</span>\nu0: (1.0, 2.0)</pre>\n```\n:::\n\n:::\n:::\n\n\nAnd\n\n::: {.cell execution_count=10}\n``` {.julia .cell-code}\nsolve(prob, Bisection())\n```\n\n::: {.cell-output .cell-output-display execution_count=11}\n```\nu: 1.1673039782614185\n```\n:::\n:::\n\n\n---\n\n\nIncorporating parameters is readily done. For example to solve $f(x) = \\cos(x) - x/p$ for different values of $p$ we might have:\n\n::: {.cell execution_count=11}\n``` {.julia .cell-code}\nf(x, p) = @. cos(x) - x/p\nu0 = (0, pi/2)\np = 2\nprob = NonlinearProblem(f, u0, p)\nsolve(prob, Bisection())\n```\n\n::: {.cell-output .cell-output-display execution_count=12}\n```\nu: 1.0298665293222586\n```\n:::\n:::\n\n\n:::{.callout-note}\n## Note\nThe *insignificant* difference in stopping criteria used by `NonlinearSolve` and `Roots` is illustrated in this example, where the value returned by `NonlinearSolve` differs by one floating point value:\n\n:::\n\n::: {.cell execution_count=12}\n``` {.julia .cell-code}\nan = solve(NonlinearProblem{false}(f, u0, p), Bisection())\nar = solve(Roots.ZeroProblem(f, u0), Roots.Bisection(); p=p)\nnextfloat(an[]) == ar, f(an[], p), f(ar, p)\n```\n\n::: {.cell-output .cell-output-display execution_count=13}\n```\n(true, 2.220446049250313e-16, 0.0)\n```\n:::\n:::\n\n\n---\n\nWe can solve for several parameters at once, by using an equal number of initial positions as follows:\n\n::: {.cell execution_count=13}\n``` {.julia .cell-code}\nps = [1, 2, 3, 4]\nu0 = @SVector[1, 1, 1, 1]\nprob = NonlinearProblem(f, u0, ps)\nsolve(prob, NewtonRaphson())\n```\n\n::: {.cell-output .cell-output-display execution_count=14}\n```\nu: 4-element SVector{4, Float64} with indices SOneTo(4):\n 0.7390851332151607\n 1.0298665293222589\n 1.1701209500026262\n 1.2523532340025887\n```\n:::\n:::\n\n\n### Higher dimensions\n\n\nWe solve now for a point on the surface of the following `peaks` function where the gradient is $0$. (The gradient here will be a vector-valued function from $R^2$ to $R^2.$) First we define the function:\n\n::: {.cell execution_count=14}\n``` {.julia .cell-code}\nfunction _peaks(x, y)\n p = 3 * (1 - x)^2 * exp(-x^2 - (y + 1)^2)\n p -= 10 * (x / 5 - x^3 - y^5) * exp(-x^2 - y^2)\n p -= 1/3 * exp(-(x + 1)^2 - y^2)\n p\nend\npeaks(u) = _peaks(u[1], u[2]) # pass container, take first two components\n```\n\n::: {.cell-output .cell-output-display execution_count=15}\n```\npeaks (generic function with 1 method)\n```\n:::\n:::\n\n\nThe gradient can be computed different ways within `Julia`, but here we use the fact that the `ForwardDiff` package is loaded by `NonlinearSolve`. Once the function is defined, the pattern is similar to above. We provide a starting point, create a problem, then solve:\n\n::: {.cell execution_count=15}\n``` {.julia .cell-code}\n∇peaks(x, p=nothing) = NonlinearSolve.ForwardDiff.gradient(peaks, x)\nu0 = @SVector[1.0, 1.0]\nprob = NonlinearProblem(∇peaks, u0)\nu = solve(prob, NewtonRaphson())\n```\n\n::: {.cell-output .cell-output-display execution_count=16}\n```\nu: 2-element SVector{2, Float64} with indices SOneTo(2):\n 1.098272834335093\n 0.8544609795247524\n```\n:::\n:::\n\n\nWe can see that this identified value is a \"zero\" through:\n\n::: {.cell execution_count=16}\n``` {.julia .cell-code}\n∇peaks(u.u)\n```\n\n::: {.cell-output .cell-output-display execution_count=17}\n```\n2-element SVector{2, Float64} with indices SOneTo(2):\n -8.257283745649602e-16\n -6.951904329977054e-16\n```\n:::\n:::\n\n\n### Using Modeling toolkit to model the non-linear problem\n\n\nNonlinear problems can also be approached symbolically using the `ModelingToolkit` package. There is one additional step necessary.\n\n\nAs an example, we look to solve numerically for the zeros of $x^5-x-\\alpha$ for a parameter $\\alpha$. We can describe this equation as follows:\n\n::: {.cell execution_count=17}\n``` {.julia .cell-code}\nusing ModelingToolkit\n\n@variables x\n@parameters α\n\neq = x^5 - x - α ~ 0\n```\n\n::: {.cell-output .cell-output-display execution_count=18}\n```\nx^5 - x - α ~ 0\n```\n:::\n:::\n\n\nThe extra step is to specify a \"`NonlinearSystem`.\" It is a system, as in practice one or more equations can be considered. The `NonlinearSystem`constructor handles the details where the equation, the variable, and the parameter are specified. Below this is done using vectors with just one element:\n\n::: {.cell execution_count=18}\n``` {.julia .cell-code}\nns = NonlinearSystem([eq], [x], [α], name=:ns)\n```\n\n::: {.cell-output .cell-output-display execution_count=19}\n\n::: {.ansi-escaped-output}\n```{=html}\n<pre><span class=\"ansi-bold\">Model ns with 1 </span><span class=\"ansi-bold\">equations</span>\n<span class=\"ansi-bold\">States (1):</span>\n x\n<span class=\"ansi-bold\">Parameters (1):</span>\n α</pre>\n```\n:::\n\n:::\n:::\n\n\nThe `name` argument is special. The name of the object (`ns`) is assigned through `=`, but the system must also know this same name. However, the name on the left is not known when the name on the right is needed, so it is up to the user to keep them synchronized. The `@named` macro handles this behind the scenes by simply rewriting the syntax of the assignment:\n\n::: {.cell execution_count=19}\n``` {.julia .cell-code}\n@named ns = NonlinearSystem([eq], [x], [α])\n```\n\n::: {.cell-output .cell-output-display execution_count=20}\n\n::: {.ansi-escaped-output}\n```{=html}\n<pre><span class=\"ansi-bold\">Model ns with 1 </span><span class=\"ansi-bold\">equations</span>\n<span class=\"ansi-bold\">States (1):</span>\n x\n<span class=\"ansi-bold\">Parameters (1):</span>\n α</pre>\n```\n:::\n\n:::\n:::\n\n\nWith the system defined, we can pass this to `NonlinearProblem`, as was done with a function. The parameter is specified here, and in this case is `α => 1.0`. The initial guess is `[1.0]`:\n\n::: {.cell execution_count=20}\n``` {.julia .cell-code}\nprob = NonlinearProblem(ns, [1.0], [α => 1.0])\n```\n\n::: {.cell-output .cell-output-display execution_count=21}\n\n::: {.ansi-escaped-output}\n```{=html}\n<pre><span class=\"ansi-cyan-fg\">NonlinearProblem</span> with uType <span class=\"ansi-cyan-fg\">Vector{Float64}</span>. In-place: <span class=\"ansi-cyan-fg\">true</span>\nu0: 1-element Vector{Float64}:\n 1.0</pre>\n```\n:::\n\n:::\n:::\n\n\nThe problem is solved as before:\n\n::: {.cell execution_count=21}\n``` {.julia .cell-code}\nsolve(prob, NewtonRaphson())\n```\n\n::: {.cell-output .cell-output-stderr}\n```\n┌ Warning: `vendor()` is deprecated, use `BLAS.get_config()` and inspect the output instead\n│ caller = (::NonlinearSolve.DefaultLinSolve)(x::Vector{Float64}, A::Matrix{Float64}, b::Vector{Float64}, update_matrix::Bool; tol::Nothing, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) at utils.jl:124\n└ @ NonlinearSolve /Users/verzani/.julia/packages/NonlinearSolve/hDIt1/src/utils.jl:124\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=22}\n```\nu: 1-element Vector{Float64}:\n 1.1673040828230086\n```\n:::\n:::\n\n\n## Optimization (`Optimization.jl`)\n\n\nWe describe briefly the `Optimization` package which provides a common interface to *numerous* optimization packages in the `Julia` ecosystem. We discuss only the interface for `Optim.jl` defined in `OptimizationOptimJL`.\n\n\nWe begin with a simple example from first semester calculus:\n\n\n> Among all rectangles of fixed perimeter, find the one with the *maximum* area.\n\n\n\nIf the perimeter is taken to be $25$, the mathematical setup has a constraint ($P=25=2x+2y$) and an objective ($A=xy$) to maximize. In this case, the function to *maximize* is $A(x) = x \\cdot (25-2x)/2$. This is easily done different ways, such as finding the one critical point and identifying this as the point of maximum.\n\n\nTo do this last step using `Optimization` we would have.\n\n::: {.cell execution_count=22}\n``` {.julia .cell-code}\nheight(x) = @. (25 - 2x)/2\nA(x, p=nothing) = @.(- x * height(x))\n```\n\n::: {.cell-output .cell-output-display execution_count=23}\n```\nA (generic function with 2 methods)\n```\n:::\n:::\n\n\nThe minus sign is needed here as optimization routines find *minimums*, not maximums.\n\n\nTo use `Optimization` we must load the package **and** the underlying backend glue code we intend to use:\n\n``` {.julia .cell-code}\nusing Optimization\nusing OptimizationOptimJL\n```\n\n\nNext, we define an optimization function with information on how its derivatives will be taken. The following uses `ForwardDiff`, which is a good choice in the typical calculus setting, where there are a small number of inputs (just $1$ here.)\n\n::: {.cell execution_count=24}\n``` {.julia .cell-code}\nF = OptimizationFunction(A, Optimization.AutoForwardDiff())\nx0 = [4.0]\nprob = OptimizationProblem(F, x0)\n```\n\n::: {.cell-output .cell-output-display execution_count=25}\n\n::: {.ansi-escaped-output}\n```{=html}\n<pre><span class=\"ansi-cyan-fg\">OptimizationProblem</span>. In-place: <span class=\"ansi-cyan-fg\">true</span>\nu0: 1-element Vector{Float64}:\n 4.0</pre>\n```\n:::\n\n:::\n:::\n\n\nThe problem is solved through the common interface with a specified method, in this case `Newton`:\n\n::: {.cell execution_count=25}\n``` {.julia .cell-code}\nsoln = solve(prob, Newton())\n```\n\n::: {.cell-output .cell-output-display execution_count=26}\n```\nu: 1-element Vector{Float64}:\n 6.25\n```\n:::\n:::\n\n\n:::{.callout-note}\n## Note\nWe use `Newton` not `NewtonRaphson` as above. Both methods are similar, but they come from different uses – for latter for solving non-linear equation(s), the former for solving optimization problems.\n\n:::\n\nThe solution is an object containing the identified answer and more. To get the value, use index notation:\n\n::: {.cell execution_count=26}\n``` {.julia .cell-code}\nsoln[]\n```\n\n::: {.cell-output .cell-output-display execution_count=27}\n```\n6.25\n```\n:::\n:::\n\n\nThe corresponding $y$ value and area are found by:\n\n::: {.cell execution_count=27}\n``` {.julia .cell-code}\nxstar = soln[]\nheight(xstar), A(xstar)\n```\n\n::: {.cell-output .cell-output-display execution_count=28}\n```\n(6.25, -39.0625)\n```\n:::\n:::\n\n\nThe `minimum` property also holds the identified minimum:\n\n::: {.cell execution_count=28}\n``` {.julia .cell-code}\nsoln.minimum # compare with A(soln[], nothing)\n```\n\n::: {.cell-output .cell-output-display execution_count=29}\n```\n-39.0625\n```\n:::\n:::\n\n\nThe package is a wrapper around other packages. The output of the underlying package is presented in the `original` property:\n\n::: {.cell execution_count=29}\n``` {.julia .cell-code}\nsoln.original\n```\n\n::: {.cell-output .cell-output-display execution_count=30}\n```\n * Status: success\n\n * Candidate solution\n Final objective value: -3.906250e+01\n\n * Found with\n Algorithm: Newton's Method\n\n * Convergence measures\n |x - x'| = 2.25e+00 ≰ 0.0e+00\n |x - x'|/|x'| = 3.60e-01 ≰ 0.0e+00\n |f(x) - f(x')| = 5.06e+00 ≰ 0.0e+00\n |f(x) - f(x')|/|f(x')| = 1.30e-01 ≰ 0.0e+00\n |g(x)| = 0.00e+00 ≤ 1.0e-08\n\n * Work counters\n Seconds run: 1 (vs limit Inf)\n Iterations: 1\n f(x) calls: 2\n ∇f(x) calls: 2\n ∇²f(x) calls: 1\n```\n:::\n:::\n\n\n---\n\n\nThis problem can also be approached symbolically, using `ModelingToolkit`.\n\n\nFor example, we set up the problem with:\n\n::: {.cell execution_count=30}\n``` {.julia .cell-code}\nusing ModelingToolkit\n@parameters P\n@variables x\ny = (P - 2x)/2\nArea = - x*y\n```\n\n::: {.cell-output .cell-output-display execution_count=31}\n```\n-x*((1//2)*P - x)\n```\n:::\n:::\n\n\nThe above should be self explanatory. To put into a form to pass to `solve` we define a \"system\" by specifying our objective function, the variables, and the parameters.\n\n::: {.cell execution_count=31}\n``` {.julia .cell-code}\n@named sys = OptimizationSystem(Area, [x], [P])\n```\n\n::: {.cell-output .cell-output-display execution_count=32}\n\n::: {.ansi-escaped-output}\n```{=html}\n<pre><span class=\"ansi-bold\">Model sys</span>\n<span class=\"ansi-bold\">States (1):</span>\n x\n<span class=\"ansi-bold\">Parameters (1):</span>\n P</pre>\n```\n:::\n\n:::\n:::\n\n\n(This step is different, as before an `OptimizationFunction` was defined; we use `@named`, as above, to ensure the system has the same name as the identifier, `sys`.)\n\n\nThis system is passed to `OptimizationProblem` along with a specification of the initial condition ($x=4$) and the perimeter ($P=25$). A vector of pairs is used below:\n\n::: {.cell execution_count=32}\n``` {.julia .cell-code}\nprob = OptimizationProblem(sys, [x => 4.0], [P => 25.0]; grad=true, hess=true)\n```\n\n::: {.cell-output .cell-output-display execution_count=33}\n\n::: {.ansi-escaped-output}\n```{=html}\n<pre><span class=\"ansi-cyan-fg\">OptimizationProblem</span>. In-place: <span class=\"ansi-cyan-fg\">true</span>\nu0: 1-element Vector{Float64}:\n 4.0</pre>\n```\n:::\n\n:::\n:::\n\n\nThe keywords `grad=true` and `hess=true` instruct for automatic derivatives to be taken as needed. These are needed in the choice of method, `Newton`, below.\n\n\nSolving this problem then follows the same pattern as before, again with `Newton` we have:\n\n::: {.cell execution_count=33}\n``` {.julia .cell-code}\nsolve(prob, Newton())\n```\n\n::: {.cell-output .cell-output-display execution_count=34}\n```\nu: 1-element Vector{Float64}:\n 6.25\n```\n:::\n:::\n\n\n(A derivative-free method like `NelderMead()` could be used and then the `grad` and `hess` keywords above would be unnecessary, though not harmful.)\n\n\n---\n\nThe related calculus problem:\n\n\n> Among all rectangles with a fixed area, find the one with *minimum* perimeter\n\n\n\ncould be similarly approached:\n\n::: {.cell execution_count=34}\n``` {.julia .cell-code}\n@parameters Area\n@variables x\ny = Area/x # from A = xy\nP = 2x + 2y\n@named sys = OptimizationSystem(P, [x], [Area])\n\nu0 = [x => 4.0]\np = [Area => 25.0]\n\nprob = OptimizationProblem(sys, u0, p; grad=true, hess=true)\nsoln = solve(prob, LBFGS())\n```\n\n::: {.cell-output .cell-output-display execution_count=35}\n```\nu: 1-element Vector{Float64}:\n 5.000000000274741\n```\n:::\n:::\n\n\nWe used an initial guess of $x=4$ above. The `LBFGS` method is a computationally efficient modification of the Broyden-Fletcher-Goldfarb-Shanno algorithm ... It is a quasi-Newton method that updates an approximation to the Hessian using past approximations as well as the gradient.\" On this problem it performs similarly to `Newton`, though in general may be preferable for higher-dimensional problems.\n\n\n### Two dimensional\n\n\nScalar functions of two input variables can have their minimum value identified in the same manner using `Optimization.jl`.\n\n\nFor example, consider the function\n\n\n\n$$\nf(x,y) = (x + 2y - 7)^2 + (2x + y - 5)^2\n$$\n\n\nWe wish to minimize this function.\n\n\nWe begin by defining a function in `Julia`:\n\n::: {.cell execution_count=35}\n``` {.julia .cell-code}\nfunction f(u, p)\n x, y = u\n (x + 2y - 7)^2 + (2x + y - 5)^2\nend\n```\n\n::: {.cell-output .cell-output-display execution_count=36}\n```\nf (generic function with 1 method)\n```\n:::\n:::\n\n\nWe turn this into an optimization function by specifying how derivatives will be taken, as we will the `LBFGS` algorithm below:\n\n::: {.cell execution_count=36}\n``` {.julia .cell-code}\nff = OptimizationFunction(f, Optimization.AutoForwardDiff())\n```\n\n::: {.cell-output .cell-output-display execution_count=37}\n```\n(::OptimizationFunction{true, Optimization.AutoForwardDiff{nothing}, typeof(f), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}) (generic function with 1 method)\n```\n:::\n:::\n\n\nWe will begin our search at the origin. We have to be mindful to use floating point numbers here:\n\n::: {.cell execution_count=37}\n``` {.julia .cell-code}\nu0 = [0.0, 0.0] # or zeros(2)\n```\n\n::: {.cell-output .cell-output-display execution_count=38}\n```\n2-element Vector{Float64}:\n 0.0\n 0.0\n```\n:::\n:::\n\n\n::: {.cell execution_count=38}\n``` {.julia .cell-code}\nprob = OptimizationProblem(ff, u0)\n```\n\n::: {.cell-output .cell-output-display execution_count=39}\n\n::: {.ansi-escaped-output}\n```{=html}\n<pre><span class=\"ansi-cyan-fg\">OptimizationProblem</span>. In-place: <span class=\"ansi-cyan-fg\">true</span>\nu0: 2-element Vector{Float64}:\n 0.0\n 0.0</pre>\n```\n:::\n\n:::\n:::\n\n\nFinally, we solve the values:\n\n::: {.cell execution_count=39}\n``` {.julia .cell-code}\nsolve(prob, LBFGS())\n```\n\n::: {.cell-output .cell-output-display execution_count=40}\n```\nu: 2-element Vector{Float64}:\n 1.0000000000000004\n 3.0\n```\n:::\n:::\n\n\nThe value of $(1, 3)$ agrees with the contour graph, as it is a point in the interior of the contour for the smallest values displayed.\n\n::: {.cell execution_count=40}\n``` {.julia .cell-code}\nusing Plots\n\nxs = range(0, 2, length=100)\nys = range(2, 4, length=100)\ncontour(xs, ys, (x,y) -> f((x,y), nothing))\n```\n\n::: {.cell-output .cell-output-display execution_count=41}\n{}\n:::\n:::\n\n\nWe could also use a *derivative-free* method, and skip a step:\n\n::: {.cell execution_count=41}\n``` {.julia .cell-code}\nprob = OptimizationProblem(f, u0) # skip making an OptimizationFunction\nsolve(prob, NelderMead())\n```\n\n::: {.cell-output .cell-output-display execution_count=42}\n```\nu: 2-element Vector{Float64}:\n 1.0000084245371434\n 2.9999988246128764\n```\n:::\n:::\n\n\n## Integration (`Integrals.jl`)\n\n\nThe `Integrals` package provides a common interface to different numeric integration packages in the `Julia` ecosystem. For example, `QuadGK` and `HCubature`. The value of this interface, over those two packages, is its non-differentiated access to other packages, which for some uses may be more performant.\n\n\nThe package follows the same `problem-algorithm-solve` interface, as already seen.\n\n\nThe interface is designed for $1$-and-higher dimensional integrals.\n\n\nThe package is loaded with\n\n``` {.julia .cell-code}\nusing Integrals\n```\n\n\nFor a simple definite integral, such as $\\int_0^\\pi \\sin(x)dx$, we have:\n\n::: {.cell execution_count=43}\n``` {.julia .cell-code}\nf(x, p) = sin(x)\nprob = IntegralProblem(f, 0.0, pi)\nsoln = solve(prob, QuadGKJL())\n```\n\n::: {.cell-output .cell-output-display execution_count=44}\n```\nu: 2.0\n```\n:::\n:::\n\n\nTo get access to the answer, we can use indexing notation:\n\n::: {.cell execution_count=44}\n``` {.julia .cell-code}\nsoln[]\n```\n\n::: {.cell-output .cell-output-display execution_count=45}\n```\n2.0\n```\n:::\n:::\n\n\nComparing to just using `QuadGK`, the same definite integral would be estimated with:\n\n::: {.cell execution_count=45}\n``` {.julia .cell-code}\nusing QuadGK\nquadgk(sin, 0, pi)\n```\n\n::: {.cell-output .cell-output-display execution_count=46}\n```\n(2.0, 1.7905676941154525e-12)\n```\n:::\n:::\n\n\nThe estimated upper bound on the error from `QuadGK`, is available through the `resid` property on the `Integrals` output:\n\n::: {.cell execution_count=46}\n``` {.julia .cell-code}\nsoln.resid\n```\n\n::: {.cell-output .cell-output-display execution_count=47}\n```\n1.7905676941154525e-12\n```\n:::\n:::\n\n\nThe `Integrals` solution is a bit more verbose, but it is more flexible. For example, the `HCubature` package provides a similar means to compute $n$- dimensional integrals. For this problem, the modifications would be:\n\n::: {.cell execution_count=47}\n``` {.julia .cell-code}\nf(x, p) = sin.(x)\nprob = IntegralProblem(f, [0.0], [pi])\nsoln = solve(prob, HCubatureJL())\n```\n\n::: {.cell-output .cell-output-display execution_count=48}\n```\nu: 1-element SVector{1, Float64} with indices SOneTo(1):\n 2.0000000000000004\n```\n:::\n:::\n\n\n::: {.cell execution_count=48}\n``` {.julia .cell-code}\nsoln[]\n```\n\n::: {.cell-output .cell-output-display execution_count=49}\n```\n2.0000000000000004\n```\n:::\n:::\n\n\nThe estimated maximum error is also given by `resid`:\n\n::: {.cell execution_count=49}\n``` {.julia .cell-code}\nsoln.resid\n```\n\n::: {.cell-output .cell-output-display execution_count=50}\n```\n1.7896795156957523e-12\n```\n:::\n:::\n\n\n---\n\n\nAs well, suppose we wanted to parameterize our function and then differentiate.\n\n\nConsider $d/dp \\int_0^\\pi \\sin(px) dx$. We can do this integral directly to get\n\n\n\n$$\n\\begin{align*}\n\\frac{d}{dp} \\int_0^\\pi \\sin(px)dx\n&= \\frac{d}{dp}\\left( \\frac{-1}{p} \\cos(px)\\Big\\rvert_0^\\pi\\right)\\\\\n&= \\frac{d}{dp}\\left( -\\frac{\\cos(p\\cdot\\pi)-1}{p}\\right)\\\\\n&= \\frac{\\cos(p\\cdot \\pi) - 1)}{p^2} + \\frac{\\pi\\cdot\\sin(p\\cdot\\pi)}{p}\n\\end{align*}\n$$\n\n\nUsing `Integrals` with `QuadGK` we have:\n\n::: {.cell execution_count=50}\n``` {.julia .cell-code}\nf(x, p) = sin(p*x)\nfunction ∫sinpx(p)\n prob = IntegralProblem(f, 0.0, pi, p)\n solve(prob, QuadGKJL())\nend\n```\n\n::: {.cell-output .cell-output-display execution_count=51}\n```\n∫sinpx (generic function with 1 method)\n```\n:::\n:::\n\n\nWe can compute values at both $p=1$ and $p=2$:\n\n::: {.cell execution_count=51}\n``` {.julia .cell-code}\n∫sinpx(1), ∫sinpx(2)\n```\n\n::: {.cell-output .cell-output-display execution_count=52}\n```\n(fill(2.0), fill(-1.6146076008187368e-16))\n```\n:::\n:::\n\n\nTo find the derivative in $p$ , we have:\n\n::: {.cell execution_count=52}\n``` {.julia .cell-code}\nForwardDiff.derivative(∫sinpx, 1), ForwardDiff.derivative(∫sinpx, 2)\n```\n\n::: {.cell-output .cell-output-display execution_count=53}\n```\n(fill(-1.9999999999999993), fill(-6.975736996017264e-16))\n```\n:::\n:::\n\n\n(In `QuadGK`, the following can be differentiated `∫sinpx(p) = quadgk(x -> sin(p*x), 0, pi)[1]` as well. `Integrals` gives a consistent interface.\n\n\n### Higher dimension integrals\n\n\nThe power of a common interface is the ability to swap backends and the uniformity for different dimensions. Here we discuss integrals of scalar-valued and vector-valued functions.\n\n\n#### $f: R^n \\rightarrow R$\n\n\nThe area under a surface generated by $z=f(x,y)$ over a rectangular region $[a,b]\\times[c,d]$ can be readily computed. The two coding implementations require $f$ to be expressed as a function of a vector–*and* a parameter–and the interval to be expressed using two vectors, one for the left endpoints (`[a,c]`) and on for the right endpoints (`[b,d]`).\n\n\nFor example, the area under the function $f(x,y) = 1 + x^2 + 2y^2$ over $[-1/2, 1/2] \\times [-1,1]$ is computed by:\n\n::: {.cell execution_count=53}\n``` {.julia .cell-code}\nf(x, y) = 1 + x^2 + 2y^2 # match math\nfxp(x, p) = f(x[1], x[2]) # prepare for IntegralProblem\nls = [-1/2, -1] # left endpoints\nrs = [1/2, 1] # right endpoints\nprob = IntegralProblem(fxp, ls, rs)\nsoln = solve(prob, HCubatureJL())\n```\n\n::: {.cell-output .cell-output-display execution_count=54}\n```\nu: 3.5\n```\n:::\n:::\n\n\nOf course, we could have directly defined the function (`fxp`) using indexing of the `x` variable.\n\n\n---\n\nFor non-rectangular domains a change of variable is required.\n\n\nFor example, an integral to assist in finding the volume of a sphere might be\n\n\n\n$$\nV = 2 \\iint_R \\sqrt{\\rho^2 - x^2 - y^2} dx dy\n$$\n\n\nwhere $R$ is the disc of radius $\\rho$ in the $x-y$ plane.\n\n\nThe usual approach is to change to polar-coordinates and write this integral as\n\n\n\n$$\nV = \\int_0^{2\\pi}\\int_0^\\rho \\sqrt{\\rho^2 - r^2} r dr d\\theta\n$$\n\n\nthe latter being an integral over a rectangular domain.\n\n\nTo compute this transformed integral, we might have:\n\n::: {.cell execution_count=54}\n``` {.julia .cell-code}\nfunction vol_sphere(ρ)\n f(rθ, p) = sqrt(ρ^2 - rθ[1]^2) * rθ[1]\n ls = [0,0]\n rs = [ρ, 2pi]\n prob = IntegralProblem(f, ls, rs)\n solve(prob, HCubatureJL())\nend\n\nvol_sphere(2)\n```\n\n::: {.cell-output .cell-output-display execution_count=55}\n```\nu: 16.75516082024234\n```\n:::\n:::\n\n\nIf it is possible to express the region to integrate as $G(R)$ where $R$ is a rectangular region, then the change of variables formula,\n\n\n\n$$\n\\iint_{G(R)} f(x) dA = \\iint_R (f\\circ G)(u) |det(J_G(u)| dU\n$$\n\n\nturns the integral into the non-rectangular domain $G(R)$ into one over the rectangular domain $R$. The key is to *identify* $G$ and to compute the Jacobian. The latter is simply accomplished with `ForwardDiff.jacobian`.\n\n\nFor an example, we find the moment of inertia about the axis of the unit square tilted counter-clockwise an angle $0 \\leq \\alpha \\leq \\pi/2$.\n\n\nThe counter clockwise rotation of a unit square by angle $\\alpha$ is described by\n\n\n\n$$\nG(u, v) = \\langle \\cos(\\alpha)\\cdot u - \\sin(\\alpha)\\cdot v, \\sin(\\alpha)\\cdot u, +\\cos(\\alpha)\\cdot v \\rangle\n$$\n\n\nSo we have $\\iint_{G(R)} x^2 dA$ is computed by the following with $\\alpha=\\pi/4$:\n\n::: {.cell execution_count=55}\n``` {.julia .cell-code}\nimport LinearAlgebra: det\n\n\n𝑓(uv) = uv[1]^2\n\nfunction G(uv)\n\n α = pi/4 # could be made a parameter\n\n u,v = uv\n [cos(α)*u - sin(α)*v, sin(α)*u + cos(α)*v]\nend\n\nf(u, p) = (𝑓∘G)(u) * det(ForwardDiff.jacobian(G, u))\n\nprob = IntegralProblem(f, [0,0], [1,1])\nsolve(prob, HCubatureJL())\n```\n\n::: {.cell-output .cell-output-display execution_count=56}\n```\nu: 0.08333333333333331\n```\n:::\n:::\n\n\n#### $f: R^n \\rightarrow R^m$\n\n\nThe `Integrals` package provides an interface for vector-valued functions. By default, the number of dimensions in the output is assumed to be $1$, but the `nout` argument can adjust that.\n\n\nLet $f$ be vector valued with components $f_1, f_2, \\dots, f_m$, then the output below is the vector with components $\\iint_R f_1 dV, \\iint_R f_2 dV, \\dots, \\iint_R f_m dV$.\n\n\nFor a trivial example, we have:\n\n::: {.cell execution_count=56}\n``` {.julia .cell-code}\nf(x, p) = [x[1], x[2]^2]\nprob = IntegralProblem(f, [0,0],[3,4], nout=2)\nsolve(prob, HCubatureJL())\n```\n\n::: {.cell-output .cell-output-display execution_count=57}\n```\nu: 2-element Vector{Float64}:\n 18.0\n 63.999999999999986\n```\n:::\n:::\n\n\n",
|
||
"supporting": [
|
||
"SciML_files"
|
||
],
|
||
"filters": [],
|
||
"includes": {}
|
||
}
|
||
} |