daviddoji/CalculusWithJuliaNotes.jl
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

lots of cleanup

Browse Source
This commit is contained in:
jverzani
2026-08-11 17:17:08 -04:00
parent ae461659e0
commit 253295ff6e
91 changed files with 18284 additions and 7872 deletions
Download Patch File Download Diff File Expand all files Collapse all files

772
quarto/derivatives/more_zeros.qmd
View File

@@ -1,5 +1,4 @@
# Derivative-free alternatives to Newton's method
# Other zero-finding algorithms
{{< include ../_common_code.qmd >}}
@@ -17,379 +16,537 @@ using SymPy
---
Newton's method is not the only algorithm of its kind for identifying zeros of a function. In this section we discuss some alternatives.
There are *numerous* zero-finding methods in addition to the secant method and Newton's method. This section shows a few different directions. It then discusses the topic of when to terminate an algorithm.
This section is entirely optional, none of the algorithms discussed below are utilized in the sequel.
## Other methods
We discuss variations of both Newton's method and the bisection method.
### Estimating the derivative
[Sidi](https://www.math.nthu.edu.tw/~amen/2008/070227-1.pdf) starts with Newton's method with its update step
$$
x_{i+1} = x_i - \frac{f(x_i)}{f'(x_i)}
$$
and notes that the secant method just uses the slope of the secant line between $x_i$ and $x_{i-1}$ to estimate $f'(x_i)$. The secant line is the *linear* polynomial interpolating the two points $(x_i, f(x_i))$ and $(x_{i-1}, f(x_{i-1}))$ and the slope the derivative of this polynomial. Sidi generalizes this to approximate the function $f'(x_i)$ by using more points from the algorithm to interpolate a polynomial at $x_i, x_{i-1}, \dots, x_{i-k}$ and then using the derivative of this polynomial to estimate the derivative of $f$ at $x_i$.
## The `find_zero(f, x0)` function
Let's consider a case with three points or $k=2$. Here we use formula (10) of Sidi computing the derivative of the interpolating polynomial at $x_n$ specialized for $k=2$.
```{julia}
function pprime_n2(xn_2, xn_1, xn, yn_2, yn_1, yn)
m = (yn - yn_1) / (xn - xn_1)
m_1 = (yn_1 - yn_2) / (xn_1 - xn_2)
m + (m - m_1)/(xn - xn_2) * (xn - xn_1)
end
```
The function `find_zero` from the `Roots` packages provides several different algorithms for finding a zero of a function, including some derivative-free algorithms for finding zeros when started with a nearby initial guess. The default method is similar to Newton's method in that only a good, initial guess is needed. However, the algorithm, while possibly slower in terms of function evaluations and steps, is engineered to be a bit more robust to the choice of initial estimate than Newton's method. (If it finds a bracket, it will use a bisection algorithm which is guaranteed to converge, but can be slower to do so.) Here we see how to call the function:
Typically, we would start with two initial points and apply the secant method to produce a third, but for this example we will use an initial three points.
```{julia}
f(x) = cos(x) - x/2
xs = [0, pi/2, pi/4]
ys = f.(xs) # y₀, y₁, y₂
xx = xs[end] - ys[end] / pprime_n2(xs[end-2:end]..., ys[end-2:end]...)
yy = f(xx)
push!(xs, xx); push!(ys, yy) # y₃
```
This method generalizes the secant method with a convergence rate of $1.83928\cdots$. For this problem we see it takes 5 iterations to converge to machine tolerance:
```{julia}
xx = xs[end] - ys[end] / pprime_n2(xs[end-2:end]..., ys[end-2:end]...)
yy = f(xx)
push!(xs, xx); push!(ys, yy) # y₄ = 0.0004182051168989398
xx = xs[end] - ys[end] / pprime_n2(xs[end-2:end]..., ys[end-2:end]...)
yy = f(xx)
push!(xs, xx); push!(ys, yy) # y₅ = -3.6161489780361933e-7
xx = xs[end] - ys[end] / pprime_n2(xs[end-2:end]..., ys[end-2:end]...)
yy = f(xx)
push!(xs, xx); push!(ys, yy) # y₆ = 3.609335053056384e-13
xx = xs[end] - ys[end] / pprime_n2(xs[end-2:end]..., ys[end-2:end]...)
yy = f(xx)
push!(xs, xx); push!(ys, yy) # y₇ = 0.0
xx
```
### Estimating the inverse function
Suppose $f^{-1}$ exists in a neighborhood of $\alpha$ and we have generated steps in our algorithm $x_0, x_1, \dots, x_n$. We can find $\alpha$ from $f^{-1}(0)$. Typically though, we wouldn't have the inverse function, but we can use facts about functions such as linearization, which *near* $0$ has:
$$
f^{-1}(y) \approx f^{-1}(0) + (f^{-1})'(0)\cdot(y) = \alpha + (f^{-1})'(0)\cdot y
$$
Solving for $\alpha$ gives for $(x_i, f(x_i))$
$$
\alpha \approx f^{-1}(f(x_i)) - f^{-1}(0) f(x_i).
$$
Replacing $f^{-1}(0) \approx (f^{-1})'(y_i) = (f^{-1})'(f(x_i)) = 1/f'(x_i)$ we get:
$$
\alpha \approx x_i - \frac{f(x_i)}{f'(x_i)}.
$$
Which is basically Newton's method.
The above uses one point, $(x_i, f(x_i))$ and the fact that $f$ is differentiable. What if two (or more) points were used, would that give some insight?
Here is some code that interpolates a *polynomial* for $f^{-1}(y)$ through $k$ points and then solves for it's value at $0$, which is $a_0$.
```{julia}
f(x) = cos(x) - x
x₀ = 1
find_zero(f, x₀)
@syms x[1:5] y[1:5] a[0:5]
a₀ = first(a)
function I(k)
eqs = Tuple(sum(a[j] * y[i]^(j-1) for j in 1:k) ~ x[i] for i in 1:k)
sols = solve(eqs, Tuple(a[1:k]))
sols[a₀] # intercept
end
```
Compare to this related call which uses the bisection method:
We use $k=2$ and see what comes:
```{julia}
find_zero(f, (0, 1)) ## [0,1] must be a bracketing interval
I(2)
```
For this example both give the same answer, but the bisection method is a bit less convenient as a bracketing interval must be pre-specified.
## The secant method
The default `find_zero` method above uses a secant-like method unless a bracketing method is found. The secant method is historic, dating back over $3000$ years. Here we discuss the secant method in a more general framework.
One way to view Newton's method is through the inverse of $f$ (assuming it exists): if $f(\alpha) = 0$ then $\alpha = f^{-1}(0)$.
If $f$ has a simple zero at $\alpha$ and is locally invertible (that is some $f^{-1}$ exists) then the update step for Newton's method can be identified with:
* fitting a polynomial to the local inverse function of $f$ going through the point $(f(x_0),x_0)$,
* and matching the slope of $f$ at the same point.
That is, we can write $g(y) = h_0 + h_1 (y-f(x_0))$. Then $g(f(x_0)) = x_0 = h_0$, so $h_0 = x_0$. From $g'(f(x_0)) = 1/f'(x_0)$, we get $h_1 = 1/f'(x_0)$. That is, $g(y) = x_0 + (y-f(x_0))/f'(x_0)$. At $y=0,$ we get the update step $x_1 = g(0) = x_0 - f(x_0)/f'(x_0)$.
A similar viewpoint can be used to create derivative-free methods.
For example, the [secant method](https://en.wikipedia.org/wiki/Secant_method) can be seen as the result of fitting a degree-$1$ polynomial approximation for $f^{-1}$ through two points $(f(x_0),x_0)$ and $(f(x_1), x_1)$.
Again, expressing this approximation as $g(y) = h_0 + h_1(y-f(x_1))$ leads to $g(f(x_1)) = x_1 = h_0$. Substituting $f(x_0)$ gives $g(f(x_0)) = x_0 = x_1 + h_1(f(x_0)-f(x_1))$. Solving for $h_1$ leads to $h_1=(x_1-x_0)/(f(x_1)-f(x_0))$. Then $x_2 = g(0) = x_1 - (x_1-x_0)/(f(x_1)-f(x_0)) \cdot f(x_1)$. This is the first step of the secant method:
$$
x_{n+1} = x_n - f(x_n) \frac{x_n - x_{n-1}}{f(x_n) - f(x_{n-1})}.
$$
That is, where the next step of Newton's method comes from the intersection of the tangent line at $x_n$ with the $x$-axis, the next step of the secant method comes from the intersection of the secant line defined by $x_n$ and $x_{n-1}$ with the $x$ axis. That is, the secant method simply replaces $f'(x_n)$ with the slope of the secant line between $x_n$ and $x_{n-1}$.
We code the update step as `λ2`:
We can see this is a rewriting of the secant method through:
```{julia}
λ2(f0,f1,x0,x1) = x1 - f1 * (x1-x0) / (f1-f0)
x1, x2 = x[1:2]; y1, y2 = y[1:2]
m = (y2 - y1) / (x2 - x1)
secant_method = x2 - (1/m) * y2
simplify(I(2) - secant_method)
```
An inverse quadratic step ($k=2$) is utilized by Brent's method, as possible, to yield a rapidly convergent bracketing algorithm implemented as a default zero finder in many software languages. `Julia`'s `Roots` package implements the method in `Roots.Brent()`.
Then we can run a few steps to identify the zero of sine starting at $3$ and $4$
To see an example of inverse quadratic, we first make a function to compute the next $x$ value, given three previous $x$ and $f(x)$ values.
```{julia}
#| hold: true
#| term: true
x0,x1 = 4,3
f0,f1 = sin.((x0,x1))
@show x1,f1
x0,x1 = x1, λ2(f0,f1,x0,x1)
f0,f1 = f1, sin(x1)
@show x1,f1
x0,x1 = x1, λ2(f0,f1,x0,x1)
f0,f1 = f1, sin(x1)
@show x1,f1
x0,x1 = x1, λ2(f0,f1,x0,x1)
f0,f1 = f1, sin(x1)
@show x1,f1
x0,x1 = x1, λ2(f0,f1,x0,x1)
f0,f1 = f1, sin(x1)
x1,f1
u = lambdify(I(3), (x[1:3]..., y[1:3]...))
```
Like Newton's method, the secant method converges quickly for this problem (though its rate is less than the quadratic rate of Newton's method).
This method is included in `Roots` as `Secant()` (or `Order1()`):
Let's try initial values $(x_0, x_1, x_2) = (0, \pi/2, \pi/4)$:
```{julia}
find_zero(sin, (4,3), Secant())
f(x) = cos(x) - x/2
xs = [0, pi/2, pi/4]
ys = f.(xs)
```
Though the derivative is related to the slope of the secant line, that is in the limit. The convergence of the secant method is not as fast as Newton's method, though at each step of the secant method, only one new function evaluation is needed, so it can be more efficient for functions that are expensive to compute or differentiate.
Now we do a step. The new values is "pushed" to the vector of values.
```{julia}
xx = u(xs[end-2:end]..., ys[end-2:end]...)
yy = f(xx)
push!(xs, xx)
push!(ys, yy)
xx, yy
```
We know do a few more steps, the value of `yy` is shown as a comment.
```{julia}
xx = u(xs[end-2:end]..., ys[end-2:end]...); yy = f(xx)
push!(xs, xx); push!(ys, yy) # yy = 0.0011053827937966831
xx = u(xs[end-2:end]..., ys[end-2:end]...); yy = f(xx)
push!(xs, xx); push!(ys, yy) # yy = -2.113895167021873e-6
xx = u(xs[end-2:end]..., ys[end-2:end]...); yy = f(xx)
push!(xs, xx); push!(ys, yy) # yy = 1.1904810470753091e-11
xx = u(xs[end-2:end]..., ys[end-2:end]...); yy = f(xx)
push!(xs, xx); push!(ys, yy) # yy = 0.0
xx
```
The convergence happens quite quickly with this well-behaved problem.
An inverse cubic interpolation is utilized by [Alefeld, Potra, and Shi](https://dl.acm.org/doi/10.1145/210089.210111) which gives an asymptotically even more rapidly convergent algorithm than Brent's (implemented in `Roots.AlefeldPotraShi()` and also `Roots.A42()`). This is used as a finishing step in many cases by the default hybrid `Order0()` method of `find_zero`.
Let $\epsilon_{n+1} = x_{n+1}-\alpha$, where $\alpha$ is assumed to be the *simple* zero of $f(x)$ that the secant method converges to. A [calculation](https://math.okstate.edu/people/binegar/4513-F98/4513-l08.pdf) shows that
$$
\begin{align*}
\epsilon_{n+1} &\approx \frac{x_n-x_{n-1}}{f(x_n)-f(x_{n-1})} \frac{(1/2)f''(\alpha)(\epsilon_n-\epsilon_{n-1})}{x_n-x_{n-1}} \epsilon_n \epsilon_{n-1}\\
& \approx \frac{f''(\alpha)}{2f'(\alpha)} \epsilon_n \epsilon_{n-1}\\
&= C \epsilon_n \epsilon_{n-1}.
\end{align*}
$$
The constant `C` is similar to that for Newton's method, and reveals potential troubles for the secant method similar to those of Newton's method: a poor initial guess (the initial error is too big), the second derivative is too large, the first derivative too flat near the answer.
Assuming the error term has the form $\epsilon_{n+1} = A|\epsilon_n|^\phi$ and substituting into the above leads to the equation
$$
\frac{A^{1+1/\phi}}{C} = |\epsilon_n|^{1 - \phi +1/\phi}.
$$
The left side being a constant suggests $\phi$ solves: $1 - \phi + 1/\phi = 0$ or $\phi^2 -\phi - 1 = 0$. The solution is the golden ratio, $(1 + \sqrt{5})/2 \approx 1.618\dots$.
### Steffensen's method
Another alternative to the secant method is Steffensen's method.
Steffensen's method is a secant-like method that converges with $|\epsilon_{n+1}| \approx C |\epsilon_n|^2$. The secant is taken between the points $(x_n,f(x_n))$ and $(x_n + f(x_n), f(x_n + f(x_n))$. Like Newton's method this requires $2$ function evaluations per step. Steffensen's is implemented through `Roots.Steffensen()`. Steffensen's method is more sensitive to the initial guess than other methods, so in practice must be used with care, though it is a starting point for many higher-order derivative-free methods.
The secant method has super-linear convergence, but not quadratic convergence. It uses these points to evaluate the values $(x_i, f(x_i)$ and $(x_{i-1}, f(x_{i-1}))$. When $x_i$ converges to $\alpha$, $x_i - x_{i-1}$ will converge to $0$. The secant lines used are eventually "converging" to tangent lines.
Steffensen's method takes a different pair of points to use for a secant line, these being $(x_n,f(x_n))$ and $(x_n + f(x_n), f(x_n + f(x_n)))$. When $x_i \rightarrow \alpha$ it follows for a continuous $f(x)$ that $f(x_i) \rightarrow 0$, so the secant lines used by Steffensen's method will also be close to the tangent line.
[This note](https://fractal.math.unr.edu/~ejolson/701-12/code/hw2sol/hw2sol.pdf) shows that with $\eta_i$ and $\xi_i$ being values that *converge* to $\alpha$, that
$$
e_{i+1} = -e_{i}^2 \cdot
\left(\frac{f''(\xi_i)\left(f'(x_i) - \frac{1}{2} f''(\eta_i) e_i\right) +f''(\eta_i)}{
2f'(x_i) + f''(\xi_i)f(x_i)}\right)
$$
As the following ratio converges to something non zero under assumptions, the Steffensen method has quadratic convergence.
$$
e_{i+1}/e_i^2 \rightarrow \frac{f''(\alpha)(1 + f'(\alpha))}{2f'(\alpha)}
$$
## Inverse quadratic interpolation
Like Newton's method this method requires $2$ function evaluations per step, but unlike Newton's method is derivative free. Steffensen's is implemented in the `Roots` package through `Roots.Steffensen()`. Steffensen's method is more sensitive to the initial guess than other methods, so in practice must be used with care, though it is a starting point for many higher-order derivative-free methods.
Inverse quadratic interpolation fits a quadratic polynomial through three points, not just two like the Secant method. The third being $(f(x_2), x_2)$.
### Alternative bracketing methods
For example, here is the inverse quadratic function, $g(y)$, going through three points marked with red dots. The blue dot is found from $(g(0), 0)$.
The bisection method has several advantages, primarily it is guaranteed to converge regardless of any assumptions on the shape of the function. It's implementation in `Roots` can handle any $x$ values as long as the function value has a sign (not `NaN` and not an error). However, it is slow---linearly convergent. There can be improvements.
#### Regula falsi
One alternative is the (modified) *regula falsi* method which replaces the midpoint ($x_i/2 + x_{i-1}/2$) with the intersection point of the line between two bracketing points $(x_i, f(x_i))$ and $(x_{i-1}, f(x_{i-1}))$ given by solving the following, which comes from similar triangles:
```{julia}
#| hold: true
@syms xᵢ xᵢ₋₁ yᵢ yᵢ₋₁ x
only(solve(yᵢ / (x - xᵢ) ~ -yᵢ₋₁ / (xᵢ₋₁ - x), x))
```
As seen earlier, this formula is a single step of the secant method, but unlike the secant method, for this method the two points chosen to continue are picked to ensure $x_i, x_{i-1}$ form a bracketing interval.
Despite being related to the secant method, the convergence rate of *regula falsi* is only linear. Some function shapes preference a certain endpoint, whereas the secant method chooses the last two values.
@fig-regula-false-convex show that some function shapes result in one end point being fixed as the algorithm progresses which can lead to linear convergence.
::: {#fig-regula-false-convex}
```{julia}
#| echo: false
a,b,c = 1,2,3
fa,fb,fc = -1,1/4,1
g(y) = (y-fb)*(y-fa)/(fc-fb)/(fc-fa)*c + (y-fc)*(y-fa)/(fb-fc)/(fb-fa)*b + (y-fc)*(y-fb)/(fa-fc)/(fa-fb)*a
ys = range(-2,2, length=100)
xs = g.(ys)
plot(xs, ys, legend=false)
scatter!([a,b,c],[fa,fb,fc], color=:red, markersize=5)
scatter!([g(0)],[0], color=:blue, markersize=5)
plot!(zero, color=:blue)
```
Here we use `SymPy` to identify the degree-$2$ polynomial as a function of $y$, then evaluate it at $y=0$ to find the next step:
```{julia}
@syms y hs[0:2] xs[0:2] fs[0:2]
H(y) = sum(hᵢ*(y - fs[end])^i for (hᵢ,i) ∈ zip(hs, 0:2))
eqs = tuple((H(fᵢ) ~ xᵢ for (xᵢ, fᵢ) ∈ zip(xs, fs))...)
ϕ = solve(eqs, hs)
hy = subs(H(y), ϕ)
```
The value of `hy` at $y=0$ yields the next guess based on the past three, and is given by:
```{julia}
q⁻¹ = hy(y => 0)
```
Though the above can be simplified quite a bit when computed by hand, here we simply make this a function with `lambdify` which we will use below.
```{julia}
λ3 = lambdify(q⁻¹) # fs, then xs
```
(`SymPy`'s `lambdify` function, by default, picks the order of its argument lexicographically, in this case they will be the `f` values then the `x` values.)
An inverse quadratic step is utilized by Brent's method, as possible, to yield a rapidly convergent bracketing algorithm implemented as a default zero finder in many software languages. `Julia`'s `Roots` package implements the method in `Roots.Brent()`. An inverse cubic interpolation is utilized by [Alefeld, Potra, and Shi](https://dl.acm.org/doi/10.1145/210089.210111) which gives an asymptotically even more rapidly convergent algorithm than Brent's (implemented in `Roots.AlefeldPotraShi()` and also `Roots.A42()`). This is used as a finishing step in many cases by the default hybrid `Order0()` method of `find_zero`.
In a bracketing algorithm, the next step should reduce the size of the bracket, so the next iterate should be inside the current bracket. However, quadratic convergence does not guarantee this to happen. As such, sometimes a substitute method must be chosen.
[Chandrapatla's](https://www.google.com/books/edition/Computational_Physics/cC-8BAAAQBAJ?hl=en&gbpv=1&pg=PA95&printsec=frontcover) method, is a bracketing method utilizing an inverse quadratic step as the centerpiece. The key insight is the test to choose between this inverse quadratic step and a bisection step. This is done in the following based on values of $\xi$ and $\Phi$ defined within:
```{julia}
function chandrapatla(f, u, v, λ; verbose=false)
a,b = promote(float(u), float(v))
fa,fb = f(a),f(b)
@assert fa * fb < 0
if abs(fa) < abs(fb)
a,b,fa,fb = b,a,fb,fa
let
gr()
Δ = 0.2
function add_line!(xs, ys, f, a)
x = (xs[1]*ys[2] - xs[2]*ys[1])/(ys[2] - ys[1])
scatter!(collect(zip(xs, ys)); marker=(5, :orange))
plot!(collect(zip(xs, ys)); line=(1, :gray50))
scatter!([(x, 0)]; marker=(5, :blue))
annotate!([(x, 0, text(a, :top))])
plot!(plt, [(x, 0), (x, Δ)]; line=(1, :gray50))
xs[2] = x
ys[2] = f(x)
end
c, fc = a, fa
f(x) = 10*log(x)/x^3
plt = plot(; empty_style..., xlims=(0.7, 3.2))
plt = plot!(plt, f, 0.8, 3)
plot!([(0.8, 0), (3.2, 0)]; line=(1, :black), arrow=true, side=:right)
maxsteps = 100
for ns in 1:maxsteps
x₀ = 2.75
x₁ = 0.85
α = 1.0
xs = [x₁, x₀]
ys = f.(xs)
Δ = abs(b-a)
m, fm = (abs(fa) < abs(fb)) ? (a, fa) : (b, fb)
ϵ = eps(m)
if Δ ≤ 2ϵ
return m
end
@show m,fm
iszero(fm) && return m
add_line!(xs, ys, f, L"x_2")
add_line!(xs, ys, f, L"x_3")
add_line!(xs, ys, f, L"x_4")
add_line!(xs, ys, f, L"x_5")
add_line!(xs, ys, f, L"x_6")
add_line!(xs, ys, f, L"x_7")
ξ = (a-b)/(c-b)
Φ = (fa-fb)/(fc-fb)
plot!(plt, [(x₀, 0), (x₀, Δ)]; line=(1, :gray50))
plot!(plt, [(x₁, 0), (x₁, Δ)]; line=(1, :gray50))
annotate!(plt, [(x₀, 0, text(L"x_0", :top)),
(x₁, 0, text(L"x_1", :top)),
(α, 0, text(L"\alpha", :left, :top))])
scatter!(plt, [(α, 0)]; marker=(5, :green))
if Φ^2 < ξ < 1 - (1-Φ)^2
xt = λ(fa,fc,fb, a,c,b) # inverse quadratic
else
xt = a + (b-a)/2
end
ft = f(xt)
isnan(ft) && break
if sign(fa) == sign(ft)
c,fc = a,fa
a,fa = xt,ft
else
c,b,a = b,a,xt
fc,fb,fa = fb,fa,ft
end
verbose && @show ns, a, fa
end
error("no convergence: [a,b] = $(sort([a,b]))")
plotly()
plt
end
```
Like bisection, this method ensures that $a$ and $b$ is a bracket, but it moves $a$ to the newest estimate, so does not maintain that $a < b$ throughout.
Plot illustrating that the *regula falsi* method may have a fixed endpoint for some convex functions
:::
We can see it in action on the sine function. Here we pass in $\lambda$, but in a real implementation (as in `Roots.Chandrapatla()`) we would have programmed the algorithm to compute the inverse quadratic value.
#### Modified regula falsi
@fig-modified-regula-falsi shows a scenario where the secant line between $(x_{i-1}, f(x_{i-1}))$ and $(x_i, f(x_i))$ crosses the $x$ axis at $x_{i+1}$ which is to the *right* of the zero $\alpha$, as it always will be for this curve and these points. A modified *regula falsi* method modifies the fixed end by using $\tilde{f}(x_i)$ and not $f(x_i)$ to compute the secant line, where $\tilde{f}$ is some multiple, $\gamma$, of $f$. In the figure, $\gamma$ is shown so that the *next* choice ($x_{i+2}$ would be its label) is exactly $\alpha$. And value for the multiplier less than this $\gamma$ will shift the intersection point to the other side of $\alpha$. The value of $\gamma$ is the ratio of the secant line slopes between $x_{i+1}$ and $\alpha$ and between $x_{i-1}$ and $\alpha$. Some choices for $\gamma$ lead to super-linear convergence.
::: {#fig-modified-regula-falsi}
```{julia}
#| echo: false
let
gr()
dd(f, a, b) = (f(a) - f(b)) / (a - b)
f(x) = (x-3)^2 - 1
plt = plot(; xlims=(0.5, 3.25), empty_style...)
plot!(plt, [(1,0), (3.25,0)]; arrow=true, side=:right, line=(1, :gray))
plot!(f, 1, 3.25; line=(1, :black))
xᵢ₋₁ , xᵢ = 1.25, 2.75
xᵢ₊₁ = (xᵢ₋₁ * f(xᵢ) - xᵢ * f( xᵢ₋₁)) / (f(xᵢ) - f(xᵢ₋₁))
α = 2
γ = dd(f, xᵢ₊₁, α) / dd(f, xᵢ₋₁, α)
plot!(plt, [(xᵢ₋₁, 0), (xᵢ₋₁, f(xᵢ₋₁))]; line=(1, :dot))
plot!(plt, [(xᵢ₋₁, f(xᵢ₋₁)) , (xᵢ, f(xᵢ))]; line=(1, :dash, :blue))
plot!(plt, [(xᵢ₋₁, γ*f(xᵢ₋₁)) , (xᵢ₊₁, f(xᵢ₊₁))]; line=(1, :dash, :blue))
scatter!(plt, [(xᵢ₋₁, γ*f(xᵢ₋₁)), (xᵢ₋₁, f(xᵢ₋₁)),
(α, 0), (xᵢ₊₁, 0),
(xᵢ₊₁, f(xᵢ₊₁)), (xᵢ, f(xᵢ))
]; marker=(3, :orange))
annotate!(plt, [
(xᵢ₋₁, 0, text(L"x_{i-1}", :top)),
(xᵢ₋₁, γ*f(xᵢ₋₁), text(L"\gamma \cdot f(x_{i-1})", :right)),
(xᵢ₋₁, f(xᵢ₋₁), text(L"f(x_{i-1})", :right)),
(α, 0, text(L"\alpha", :bottom)),
(xᵢ₊₁, 0, text(L"x_{i+1}", :top)),
(xᵢ, 0, text(L"x_{i}", :top))
])
plotly()
plt
end
```
Modified *regula falsi* method illustration. When midpoint $x_{i+1}$ is on same side of zero $\alpha$ as $x_i$ the *next* step will be between $x_{i-1}$ and $x_{i+1}$. *Were* $x_{i-1}$ modified by $\gamma$ the next midpoint would be an exact zero. If multiplied by a value less, then the midpoint moves to other side of $\alpha$ and would break the repeated choice of a fixed side when keeping a bracketing interval.
:::
#### Anderson Bjork
There are numerous modifications of the *regula falsi* algorithm that
employ different scaling values, we discuss one now. The
[Anderson-Bjork](https://iopscience.iop.org/article/10.1088/1757-899X/1276/1/012010/pdf)
method is a modification of the *regula falsi* method that avoids the
linear convergence when one endpoint is always fixed.
The modification works as follows, suppose the bracketing interval is $[a,b]$ and $c$ is the point found by the secant line. Then if $f(a)$ and $f(c)$ have the same sign **and** the previous step kept the right side point ($b$) fixed, then instead of using $(c, f(c))$ and $(b, f(c))$ as the new points (as $[c,b]$ is a bracket) use $(b, \gamma \cdot f(b))$ where $\gamma = 1 - f(c)/f(a)$ if $\gamma$ is positive and $\gamma=1/2$ if not. This will modify the next step in the algorithm. The $\gamma$ factors are multiplied each time, so that eventually the $y$ value used at the fixed side should lead to a midpoint on the other side of the zero, as happens when the modified value of $f(x_1)$ and $f(x_5)$ are used to find the midpoint in @fig-anderson-bjork-trajectory.
::: {#fig-anderson-bjork-trajectory}
```{julia}
#| echo: false
let
gr()
midpt(a,b,fa,fb) = (a*fb - b*fa)/(fb-fa)
side = nothing
function ABstep!(xs, ys, side, label)
a, b = xs; fa, fb = ys
c = midpt(a,b,fa,fb)
fc = f(c)
if sign(fa) == sign(fc)
xs[1] = c
ys[1] = fc
if side == :right
m = 1 - fc/fa
m = m < 0 ? 1/2 : m
ys[2] *= m
else
side = :right
end
else
xs[2] = c
ys[2] = fc
if side == :left
m = 1 - fc/fb
m = m < 0 ? 1/2 : m
ys[1] *= m
else
side = :left
end
end
plot!([(c,0), (c,fc)]; line=(1, :dash, :gray50))
scatter!([(c,0)]; marker=(5, :blue))
scatter!(collect(zip(xs, ys)); marker=(5, :orange))
plot!(collect(zip(xs, ys)); line=(1, :gray50))
annotate!([(c, 0, text(label, :top, :left))])
side
end
f(x) = 10*log(x)/x^3
plt = plot(f, 0.8, 3; xlims=(0.7, 3.2), empty_style...)
plot!([(0.8, 0), (3.2, 0)]; line=(1, :black), arrow=true, side=:right)
α = 1
x₀, x₁ = 2.75, 0.85
xs = [x₀, x₁]
ys = f.(xs)
scatter!(plt, collect(zip(xs, ys)); marker=(5, :orange))
plot!(plt, collect(zip(xs, ys)); line=(1, :gray50))
plot!(plt, [(x₀,0), (x₀, f(x₀))]; line=(1, :dash, :gray50))
plot!(plt, [(x₁,0), (x₁, f(x₁))]; line=(1, :dash, :gray50))
side = ABstep!(xs, ys, side, L"x_2")
side = ABstep!(xs, ys, side, L"x_3")
side = ABstep!(xs, ys, side, L"x_4")
side = ABstep!(xs, ys, side, L"x_5")
side = ABstep!(xs, ys, side, L"x_6")
c = midpt(xs..., ys...)
scatter!([(c,0)]; marker=(5, :blue))
scatter!([(α,0)]; marker=(5, :green))
annotate!([(c,0, text(L"x_7", :top, :left)),
(x₀, 0, text(L"x_0", :top, )),
(x₁, 0, text(L"x_1", :top, :right)),
(α, 0, text(L"\alpha", :top, :left))
])
plotly()
current()
end
```
Illustration of Anderson-Bjork algorithm. The point $x_1$ stays as the left-hand endpoint up until $x_6$, but by modifying $f(x_0)$ the algorithm converges super-linearly towards $\alpha$, as compared to @fig-regula-false-convex.
:::
::: {.callout-note}
## Hybrid algorithms
There are a few, newer, *hybrid algorithms* where some dynamic choice is made as to what update step should be chosen. One due to Chandrapatla (implemented in `Roots.Chandrapatla`) is a bracketing algorithm which chooses between an inverse quadratic step or a bisection step using a certain inequality. We note another, a bracketing algorithm due to Ganchovski and Traykov (with improvements by some `Julia` programmers since its inclusion in the `NonLinearSolve.jl` package) that chooses between bisection or the Anderson-Bjork update based on an estimate of how "straight" the curve is. This is implemented in `Roots.ModAB`. The latter is quite efficient over a wide range of problems.
:::
##### Examples
The function $f(x) = (x^2 + 1) \cdot \sin(x) - e^{\sqrt{\lvert x\rvert}} \cdot (x - 1) \cdot (x^2 - 5)$ has a zero *between* $0$ and $1$, and *near* $0.8$. We see how to find it using various algorithms implemented in `Roots`.
```{julia}
#| term: true
chandrapatla(sin, 3, 4, λ3, verbose=true)
f(x) = (x^2 + 1) * sin(x) - exp(sqrt(abs(x))) * (x - 1) * (x^2 - 5)
x0 = 0.8
xs = (0, 1)
```
The Steffensen method needs a *nearby* estimate:
```{julia}
find_zero(f, x0, Roots.Steffensen()) # 5 iterations, 12 function evaluations
```
The `Sidi(2)` method needs a *nearby* estimate or an initial two points for a secant line which it bootstraps to get a third point. We use the bracketing interval below:
```{julia}
find_zero(f, xs, Roots.Sidi(2)) # 3 iterations, 6 function evaluations
```
For some steps, `Brent` and `Chandrapatla` use a quadratic inverse calculation, whereas `A42` uses a cubic inverse calculation:
```{julia}
find_zero(f, xs, Roots.Brent()) # 16 iterations, 18 function evaluations
find_zero(f, xs, Roots.Chandrapatla()) # 20 iterations, 22 function evaluations
find_zero(f, xs, Roots.A42()) # 4 iterations, 10 function evaluations
```
Finally, we compare *regula falsi* variants:
```{julia}
find_zero(f, xs, Roots.RegulaFalsi(:classic)) # 10 iterations, 13 function evaluations
find_zero(f, xs, Roots.RegulaFalsi(:AndersonBjork)) # 6 iterations, 9 function evaluations
find_zero(f, xs, Roots.ModAB()) # 6 iterations, 8 function evaluations
```
For this problem, all methods converge to the same zero, but from the counts of iterations and function evaluations they differ in how the get there.
## Tolerances
Iterative zero-finding algorithms may mathematically converge, but when implemented on the computer a stopping rule must be articulated. Typically these involve the following:
The `chandrapatla` algorithm typically waits until `abs(b-a) <= 2eps(m)` (where $m$ is either $b$ or $a$ depending on the size of $f(a)$ and $f(b)$) is satisfied. Informally this means the algorithm stops when the two bracketing values are no more than a small amount apart. What is a "small amount?"
To understand, we start with the fact that floating point numbers are an approximation to real numbers.
Floating point numbers effectively represent a number in scientific notation in terms of
* a sign (plus or minus) ,
* a *mantissa* (a number in $[1,2)$, in binary ), and
* an exponent (to represent a power of $2$).
The mantissa is of the form `1.xxxxx...xxx` where there are $m$ different `x`s each possibly a `0` or `1`. The `i`th `x` indicates if the term `1/2^i` should be included in the value. The mantissa is the sum of `1` plus the indicated values of `1/2^i` for `i` in `1` to `m`. So the last `x` represents if `1/2^m` should be included in the sum. As such, the mantissa represents a discrete set of values, separated by `1/2^m`, as that is the smallest difference possible.
For example if `m=2` then the possible value for the mantissa are `11 => 1 + 1/2 + 1/4 = 7/4`, `10 => 1 + 1/2 = 6/4`, `01 => 1 + 1/4 = 5/4`. and `00 => 1 = 4/4`, values separated by `1/4 = 1/2^m`.
For $64$-bit floating point numbers `m=52`, so the values in the mantissa differ by `1/2^52 = 2.220446049250313e-16`. This is the value of `eps()`.
However, this "gap" between numbers is for values when the exponent is `0`. That is the numbers in `[1,2)`. For values in `[2,4)` the gap is twice, between `[1/2,1)` the gap is half. That is the gap depends on the size of the number. The gap between `x` and its next largest floating point number is given by `eps(x)` and that always satisfies `eps(x) <= eps() * abs(x)`.
One way to think about this is the difference between `x` and the next largest floating point values is *basically* `x*(1+eps()) - x` or `x*eps()`.
For the specific example, `abs(b-a) <= 2eps(m)` means that the gap between `a` and `b` is essentially 2 floating point values from the $x$ value with the smallest $f(x)$ value.
* stop (and fail) if too many steps are taken
* stop when $\lvert x_i - x_{i-1} \rvert$ is quite small (as then the algorithm stops improving)
* stop when $f(x_i)$ is quite small, as it is close to being zero.
Small on the computer is a *relative* term and requires a bit of discussion.
When $\lvert x_i - x_{i-1} \rvert$ is small, we have to recall that the gap between floating point numbers depends on the size of the number, and doubles in going from $[2^{i-1}, 2^i)$ to $[2^i, 2^{i+1})$. As such, a relative tolerance is often chosen so that *small* really means that for some $\epsilon$
$$
\lvert x_i - x_{i-1} \rvert \leq \max(\lvert x_i \rvert, \lvert x_{i-1} \rvert) \cdot \epsilon.
$$
In code, this might be `abs(b-a) <= 2eps(m)`, which means that the "gap" between `a` and `b` is essentially no more than $2$ floating point values from the $x$ value with the smallest $f(x)$ value.
For bracketing methods that is about as good as you can get. However, once floating point values are understood, the absolute best you can get for a bracketing interval would be
* along the way, a value `f(c)` is found which evaluates *exactly* to `0.0`
* the endpoints of the bracketing interval are *adjacent* floating point values, meaning the interval can not be bisected and `f` changes sign between the two values.
* along the way, a value `f(c)` is found which evaluates *exactly* to `0.0`
* the endpoints of the bracketing interval are *adjacent* floating point values, meaning the interval can not be bisected and `f` changes sign between the two values.
There can be problems when the stopping criteria is `abs(b-a) <= 2eps(m))` and the answer is `0.0` that require engineering around. For example, the algorithm above for the function `f(x) = -40*x*exp(-x)` does not converge when started with `[-9,1]`, even though `0.0` is an obvious zero.
There can be problems when the stopping criteria is `abs(b-a) <= 2eps(m))` and the answer is `0.0` that require engineering around. As such, an *absolute* tolerance might be needed, one where $\lvert x_i - x_{i-1} \rvert \leq \delta$.
For bracketing algorithms, consideration of $\lvert x_i - x_{i-1} \rvert$ might be all that matters, but not for algorithms like Newton's or the secant algorithm. In Newton's method the update step is $f(x_{i-1})/f'(x_{i-1})$. Naturally when $f(x_i)$ is close to $0$, the update step is small and $\lvert x_{i} - x_{i-1}\rvert = \Delta$ will be close to $0$. *However*, should $f'(x_i)$ be large, then $\Delta$ can also be small and the algorithm will possibly stop, as $x_{i} \approx x_{i-1}$---but not necessarily $x_{i} \approx \alpha$. So termination on $\Delta$ alone can be off. Checking if $f(x_{i})$ is an approximate zero---as it should be if $f$ is continuous---is also useful to include in a stopping criteria.
However, there may never be a value with `f(x_i)` exactly `0.0`. (The value of `sin(1pi)` is not zero, for example, as `1pi` is an approximation to $\pi$, as well the `sin` of values adjacent to `float(pi)` do not produce `0.0` exactly.)
```{julia}
#| hold: true
#| error: true
fu(x) = -40*x*exp(-x)
chandrapatla(fu, -9, 1, λ3)
```
Here the issue is `abs(b-a)` is tiny (of the order `1e-119`) but `eps(m)` is even smaller.
> For checking if $x_n \approx x_{n+1}$ both a relative and absolute error should be used unless something else is known.
Suppose `x_i` is the closest floating point number to $\alpha$, the mathematical zero. Then the relative rounding error, $($ `x_i` $- \alpha)/\alpha$, will be a value $\delta$ with $\delta$ less than `eps()`.
For non-bracketing methods, like Newton's method or the secant method, different criteria are useful. There may not be a bracketing interval for `f` (for example `f(x) = (x-1)^2`) so the second criteria above might need to be restated in terms of the last two iterates, $x_n$ and $x_{n-1}$. Calling this difference $\Delta = |x_n - x_{n-1}|$, we might stop if $\Delta$ is small enough. As there are scenarios where this can happen, but the function is not at a zero, a check on the size of $f$ is needed.
However, there may be no floating point value where $f$ is exactly `0.0` so checking the size of `f(x_n)` requires some agreement.
First if `f(x_n)` is `0.0` then it makes sense to call `x_n` an *exact zero* of $f$, even though this may hold even if `x_n`, a floating point value, is not mathematically an *exact* zero of $f$. (Consider `f(x) = x^2 - 2x + 1`. Mathematically, this is identical to `g(x) = (x-1)^2`, but `f(1 + eps())` is zero, while `g(1+eps())` is `4.930380657631324e-32`.
However, there may never be a value with `f(x_n)` exactly `0.0`. (The value of `sin(1pi)` is not zero, for example, as `1pi` is an approximation to $\pi$, as well the `sin` of values adjacent to `float(pi)` do not produce `0.0` exactly.)
Suppose `x_n` is the closest floating point number to $\alpha$, the zero. Then the relative rounding error, $($ `x_n` $- \alpha)/\alpha$, will be a value $\delta$ with $\delta$ less than `eps()`.
How far then can `f(x_n)` be from $0 = f(\alpha)$?
How far then can `f(x_i)` be from $0 = f(\alpha)$? Consider:
$$
f(x_n) = f(x_n - \alpha + \alpha) = f(\alpha + \alpha \cdot \delta) = f(\alpha \cdot (1 + \delta)),
f(x_i) = f(x_i - \alpha + \alpha) = f(\alpha + \alpha \cdot \delta) = f(\alpha \cdot (1 + \delta)),
$$
where $\delta = x_i/\alpha - 1$ is close to $0$ if $x_i$ converges to $\alpha$.
Assuming $f$ has a derivative, the linear approximation gives:
$$
f(x_n) \approx f(\alpha) + f'(\alpha) \cdot (\alpha\delta) = f'(\alpha) \cdot \alpha \delta
f(x_n) \approx f(\alpha) + f'(\alpha) \cdot (\alpha\delta) = \alpha \cdot f'(\alpha) \cdot \delta
$$
So we should consider `f(x_n)` an *approximate zero* when it is on the scale of $f'(\alpha) \cdot \alpha \delta$. That $\alpha$ factor means we consider a *relative* tolerance for `f`.
So we should consider `f(x_i)` an *approximate zero* when it is on the scale of $\alpha \cdot f'(\alpha) \cdot \delta$. That $\alpha$ factor means we consider a *relative* tolerance, $\delta$, for $f(x_i)$ based on $\lvert x_i\rvert$.
> For checking if $f(x_n) \approx 0$ both a relative and absolute error should be used---the relative error involving the size of $x_n$.
A good condition to check if `f(x_n)` is small is
As well though, for $\alpha$ values close to $0$ this relative tolerance might be an issue, and a small absolute tolerance can be needed.
`abs(f(x_n)) <= abs(x_n) * rtol + atol`, or `abs(f(x_n)) <= max(abs(x_n) * rtol, atol)`
A good condition to check if `f(x_i)` is small is
* `abs(f(x_i)) <= abs(x_i) * rtol + atol`, or
* `abs(f(x_i)) <= max(abs(x_i) * rtol, atol)`
where the relative tolerance, `rtol`, would absorb an estimate for $f'(\alpha)$.
Now, in Newton's method the update step is $f(x_n)/f'(x_n)$. Naturally when $f(x_n)$ is close to $0$, the update step is small and $\Delta$ will be close to $0$. *However*, should $f'(x_n)$ be large, then $\Delta$ can also be small and the algorithm will possibly stop, as $x_{n+1} \approx x_n$ – but not necessarily $x_{n+1} \approx \alpha$. So termination on $\Delta$ alone can be off. Checking if $f(x_{n+1})$ is an approximate zero is also useful to include in a stopping criteria.
One thing to keep in mind is that the right-hand side of the rule `abs(f(x_i)) <= abs(x_i) * rtol + atol`, as a function of `x_i`, goes to `Inf` as `x_i` increases. So if `f` has `0` as an asymptote (like `e^(-x)`) for large enough `x_i`, the rule will be `true` and `x_i` could be counted as an approximate zero, despite it not being one.
One thing to keep in mind is that the right-hand side of the rule `abs(f(x_n)) <= abs(x_n) * rtol + atol`, as a function of `x_n`, goes to `Inf` as `x_n` increases. So if `f` has `0` as an asymptote (like `e^(-x)`) for large enough `x_n`, the rule will be `true` and `x_n` could be counted as an approximate zero, despite it not being one.
A modified criteria for convergence might look like:
So a modified criteria for convergence might look like:
* stop if $\Delta$ is small and `f` is an approximate zero with some tolerances
* stop if $\Delta$ is small and `f` is an approximate zero with some tolerances
* stop if `f` is an approximate zero with some tolerances, but be mindful that this rule can identify mathematically erroneous answers.
* stop if `f` is an approximate zero with some tolerances, but be mindful that this rule can identify mathematically erroneous answers.
It is not uncommon to assign `rtol` to have a value like `sqrt(eps())` to account for accumulated floating point errors and the factor of $f'(\alpha)$, though in the `Roots` package it is set smaller by default.
@@ -397,12 +554,17 @@ It is not uncommon to assign `rtol` to have a value like `sqrt(eps())` to accoun
### Conditioning and stability
In Part III of @doi:10.1137/1.9781611977165 we find language of numerical analysis useful to formally describe the zero-finding problem. Key concepts are errors, conditioning, and stability. These give some theoretical justification for the tolerances above.
This next part is a technical, mathematical---not practical---motivation for why we might stop when $x_i \approx x_{i-1}$ or $f(x_i) \approx 0$.
In Part III of @doi:10.1137/1.9781611977165 we find language of numerical analysis useful to formally describe the zero-finding problem. Key concepts are errors, conditioning, and stability, which can be used to give some theoretical justification for the tolerances above.
Abstractly a *problem* is a mapping, $F$, from a domain $X$ of data to a range $Y$ of solutions. Both $X$ and $Y$ have a sense of distance given by a *norm*. A norm (denoted with $\lVert\cdot\rVert$) is a generalization of the absolute value and gives quantitative meaning to terms like small and large.
::: {.definition title="Well conditioned problem"}
> A *well-conditioned* problem is one with the property that all small perturbations of $x$ lead to only small changes in $F(x)$.
A *well-conditioned* problem is one with the property that all small perturbations of $x$ lead to only small changes in $F(x)$.
:::
This sense of "small" is measured through a *condition number*.
@@ -412,25 +574,32 @@ The *forward error* is $\lVert\delta_F\rVert = \lVert F(x+\delta_x) - F(x)\rVert
The *backward error* is $\lVert\delta_x\rVert$, the *relative backward error* is $\lVert\delta_x\rVert / \lVert x\rVert$.
The *absolute condition number* $\hat{\kappa}$ is worst case of this ratio $\lVert\delta_F\rVert/ \lVert\delta_x\rVert$ as the perturbation size shrinks to $0$.
The relative condition number $\kappa$ divides $\lVert\delta_F\rVert$ by $\lVert F(x)\rVert$ and $\lVert\delta_x\rVert$ by $\lVert x\rVert$ before taking the ratio.
The *absolute condition number*, $\hat{\kappa}$, is the worst case of the forward error divided by the backward error, or this ratio $\lVert\delta_F\rVert/ \lVert\delta_x\rVert$, as the perturbation size shrinks to $0$.
The *relative condition number*, $\kappa$, divides $\lVert\delta_F\rVert$ by $\lVert F(x)\rVert$ and $\lVert\delta_x\rVert$ by $\lVert x\rVert$ before taking the ratio.
A *problem* is a mathematical concept, an *algorithm* the computational version. Algorithms may differ for many reasons, such as floating point errors, tolerances, etc. We use notation $\tilde{F}$ to indicate the algorithm.
The absolute error in the algorithm is $\lVert\tilde{F}(x) - F(x)\rVert$, the relative error divides by $\lVert F(x)\rVert$. A good algorithm would have smaller relative errors.
The *absolute error in the algorithm* is $\lVert\tilde{F}(x) - F(x)\rVert$, the relative error divides by $\lVert F(x)\rVert$. A good algorithm would have smaller relative errors.
An algorithm is called *stable* if
$$
\frac{\lVert\tilde{F}(x) - F(\tilde{x})\rVert}{\lVert F(\tilde{x})\rVert}
\frac{\lVert\tilde{F}(x) - F(\tilde{x})\rVert}{\lVert F(\tilde{x})\rVert},
$$
is *small* for *some* $\tilde{x}$ relatively near $x$, $\lVert\tilde{x}-x\rVert/\lVert x\rVert$.
> A *stable* algorithm gives nearly the right answer to nearly the right question.
::: {.definition tilte="Stable algorithm"}
A *stable* algorithm gives nearly the right answer to nearly the right question.
:::
The right answer is $F(x)$, the nearly right answer is $F(\tilde{x})$, the nearly right question is $\tilde{F}(x)$.
(The answer it gives is $\tilde{F}(x)$, the nearly right question: what is $F(\tilde{x})$?)
A related concept is an algorithm $\tilde{F}$ for a problem $F$ is *backward stable* if for each $x \in X$,
@@ -438,9 +607,15 @@ $$
\tilde{F}(x) = F(\tilde{x})
$$
for some $\tilde{x}$ where $\lVert\tilde{x} - x\rVert/\lVert x\rVert$ is small.
for *some* $\tilde{x}$ where $\lVert\tilde{x} - x\rVert/\lVert x\rVert$ is small.
> "A backward stable algorithm gives exactly the right answer to nearly the right question."
::: {.definition tilte="Backward stable algorithm"}
"A backward stable algorithm gives exactly the right answer to nearly the right question."
:::
The nearly right question is $\tilde{F}(x)$, the exactly right answer to this is $F(\tilde{x})$.
The concepts are related by Trefethen and Bao's Theorem 15.1 which says for a backward stable algorithm the relative error $\lVert\tilde{F}(x) - F(x)\rVert/\lVert F(x)\rVert$ is small in a manner proportional to the relative condition number.
@@ -449,7 +624,7 @@ Applying this to the zero-finding we follow @doi:10.1137/1.9781611975086.
To be specific, the problem, $F$, is finding a zero of a function $f$ starting at an initial point $x_0$. The data is $(f, x_0)$, the solution is $r$ a zero of $f$.
Take the algorithm as Newton's method. Any implementation must incorporate tolerances, so this is a computational approximation to the problem. The data is the same, but technically we use $\tilde{f}$ for the function, as any computation is dependent on machine implementations. The output is $\tilde{r}$ an *approximate* zero.
For concreteness, take the algorithm as Newton's method. Any implementation must incorporate tolerances, so this is a computational approximation to the problem. The data is the same, but technically we use $\tilde{f}$ for the function, as any computation is dependent on machine implementations. The output is $\tilde{r}$ an *approximate* zero.
Suppose for sake of argument that $\tilde{f}(x) = f(x) + \epsilon$, $f$ has a continuous derivative, and $r$ is a root of $f$ and $\tilde{r}$ is a root of $\tilde{f}$. Then by linearization:
@@ -463,15 +638,22 @@ $$
$$
Rearranging gives $\lVert\delta/\epsilon\rVert \approx 1/\lVert f'(r)\rVert$. But the $|\delta|/|\epsilon|$ ratio is related to the condition number:
> The absolute condition number is $\hat{\kappa}_r = |f'(r)|^{-1}$.
::: {.definition title="Absolute condition number"}
The absolute condition number is $\hat{\kappa}_r = |f'(r)|^{-1}$.
:::
The error formula in Newton's method measuring the distance between the actual root and an approximation includes the derivative in the denominator, so we see large condition numbers are tied into possibly larger errors.
Now consider $g(x) = f(x) - f(\tilde{r})$. Call $f(\tilde{r})$ the residual. We have $g$ is near $f$ if the residual is small. The algorithm will solve $(g, x_0)$ with $\tilde{r}$, so with a small residual an exact solution to an approximate question will be found. Driscoll and Braun state
> The backward error in a root estimate is equal to the residual.
::: {.relationship title="Backward error and residual"}
The backward error in a root estimate is equal to the residual.
:::
Practically these two observations lead to
@@ -630,8 +812,8 @@ choices = [
"The function oscillates too much to rely on the tangent line approximation far from the zero",
"We can find an answer"
]
answ = 4
radioq(choices, answ, keep_order=true)
answer = 4
radioq(choices, answer, keep_order=true)
```
Does `find_zero` find a zero to this function starting from $0.175$?
@@ -664,6 +846,6 @@ choices = [
"The function oscillates too much to rely on the tangent line approximations far from the zero",
"We can find an answer"
]
answ = 3
radioq(choices, answ, keep_order=true)
answer = 3
radioq(choices, answer, keep_order=true)
```