1764 lines
50 KiB
Plaintext
1764 lines
50 KiB
Plaintext
# Newton's method
|
||
|
||
|
||
{{< include ../_common_code.qmd >}}
|
||
|
||
This section uses these add-on packages:
|
||
|
||
|
||
```{julia}
|
||
using CalculusWithJulia
|
||
using Plots
|
||
plotly()
|
||
using SymPy
|
||
using Roots
|
||
```
|
||
|
||
|
||
---
|
||
|
||
This section discusses two key algorithms for finding a zero of a real-valued function of a single variable, that is solving $f(x) = 0$.
|
||
|
||
The bisection method is one such algorithm and requires the knowledge that the zero is **between** two values, which are called a bracketing interval. The two main methods discussed here, Newton's method and the secant method, are more efficient---when they work---and usually require the knowledge of a starting point **near** the desired zero.
|
||
|
||
|
||
## The Babylonian method
|
||
|
||
We begin with a special purpose algorithm to illustrate the key ideas.
|
||
The Babylonian method is an algorithm to find an approximate value for $\sqrt{k}$. It was described by the first-century Greek mathematician Hero of [Alexandria](http://en.wikipedia.org/wiki/Babylonian_method).
|
||
|
||
The method starts with some initial guess, called $x_0$. This is usually some nearby value to the answer. The method then applies a formula to produce an improved guess. This is repeated until the improved guess is accurate enough or it is clear the algorithm fails to work.
|
||
|
||
|
||
For the Babylonian method, the next guess, $x_{i+1}$, is derived from the current guess, $x_i$ by
|
||
|
||
|
||
$$
|
||
x_{i+1} = \frac{1}{2}(x_i + \frac{k}{x_i})
|
||
$$
|
||
|
||
We use this algorithm to approximate the square root of $2$, a value known to the Babylonians.
|
||
|
||
We start with $x = 2$. In this example, we use rational numbers to keep exact quantities:
|
||
|
||
```{julia}
|
||
x₀ = 2//1
|
||
x₁ = x₀/2 + 1/x₀
|
||
```
|
||
|
||
We have $x_0^2 = 4$, what about $x_1^2$? We use a floating point exponent to see the decimal value.
|
||
|
||
```{julia}
|
||
x₁^2.0
|
||
```
|
||
|
||
A value much closer to $2$. We repeat with another step:
|
||
|
||
```{julia}
|
||
x₂ = x₁/2 + 1/x₁
|
||
x₂, x₂^2.0
|
||
```
|
||
|
||
We now see accuracy until the third decimal point. Repeating another time gives even more accuracy:
|
||
|
||
```{julia}
|
||
x₃ = x₂/2 + 1/x₂
|
||
x₃, x₃^2.0
|
||
```
|
||
|
||
Over rational numbers, the value for the estimate gets more and more complicated, as a peak at the next value shows:
|
||
|
||
```{julia}
|
||
x₄ = x₃/2 + 1/x₃
|
||
x₄, x₄^2.0
|
||
```
|
||
|
||
This is not the case over floating point numbers, where we see increasing convergence towards $\sqrt{2}$.
|
||
|
||
```{julia}
|
||
float.([x₀, x₁, x₂, x₃, x₄]) .- sqrt(2)
|
||
```
|
||
|
||
We see this algorithm rapidly converges to $\sqrt{2}$. In fact, in two more steps it will get as close as machine precision will allow a floating point number to approximate an irrational number. The algorithm produces *approximations* to the actual answer which can be easily computed in a few steps to a desired tolerance and, if needed, repeated more often to near exactness.
|
||
|
||
|
||
## Newton's method
|
||
|
||
Is there some generalization to the Babylonian method that applies to non-linear functions?
|
||
|
||
|
||
Let $f(x) = x^3 - 2x -5$. The value of $2$ is almost a zero, but not quite, as $f(2) = -1$. We can check that there are no *rational* roots. Though there is a method to solve the cubic it may be difficult to compute and will not be as generally applicable as some iterative algorithm like the Babylonian method to produce an approximate answer to a non-linear problem.
|
||
|
||
|
||
We know that the tangent line is a good approximation to the function at the point. Looking at this graph gives a hint as to an algorithm:
|
||
|
||
::: {#fig-plot-x-3-minus-2-x-minus-5}
|
||
|
||
```{julia}
|
||
#| echo: false
|
||
let
|
||
gr()
|
||
f(x) = x^3 - 2x - 5
|
||
fp(x) = 3x^2 - 2
|
||
c = 2
|
||
|
||
plt = plot(;empty_style..., xlims=(1.75, 2.25))
|
||
plot!(plt, [(1.75,0.0), (2.25,0.0)]; line=(1, :gray), arrow=true, side=:right)
|
||
|
||
plot!(plt, f; line=(2, :black))
|
||
plot!(plt, x->f(c) + fp(c)*(x-c); line=(1, :black))
|
||
scatter!(plt, [(c, f(c))]; marker=(5, :orange))
|
||
ticks = 1.8:0.1:2.2
|
||
annotate!(plt, [(x, 0.0, text(latexstring(x), :top)) for x in ticks])
|
||
annotate!([(c, f(c), text(L"(c, f(c))", :top, :left))])
|
||
for x in ticks
|
||
plot!(plt, [(x,0.0), (x, 0.1)]; line=(1, :gray))
|
||
end
|
||
plotly()
|
||
plt
|
||
end
|
||
```
|
||
|
||
|
||
|
||
Plot of $f(x) = x^3 - 2x - 5$ and a tangent line at a point, $(c, f(c))$, near a $0$
|
||
:::
|
||
|
||
The tangent line and the function nearly agree near $2$. So much so, that the intersection point of the tangent line with the $x$ axis and the intersection of $f(x)$ with the $x$ axis are nearly the same value.
|
||
|
||
The key observation is: the intersection of the tangent line and the $x$ axis should be an improved approximation for the zero of the function.
|
||
|
||
|
||
Let $x_0$ be the initial estimate for a zero, and $x_1$ be the intersection point of the tangent line at $(x_0, f(x_0))$ with the $x$ axis. Then by the definition of the tangent line:
|
||
|
||
|
||
$$
|
||
f'(x_0) = \frac{\Delta y }{\Delta x} = \frac{f(x_1) - f(x_0)}{x_1 - x_0} = \frac{0 - f(x_0)}{x_1 - x_0}.
|
||
$$
|
||
|
||
This can be solved for $x_1$ to give $x_1 = x_0 - f(x_0)/f'(x_0)$. In general, if our current approximation is $x_i$ and used the intersection point of the tangent line to produce $x_{i+1}$ we would have Newton's method:
|
||
|
||
|
||
$$
|
||
x_{i+1} = x_i - \frac{f(x_i)}{f'(x_i)}.
|
||
$$
|
||
|
||
----
|
||
|
||
Using automatic derivatives, as brought in with the `CalculusWithJulia` package, we can implement this algorithm step by step. Starting at $x_0=2$ we have:
|
||
|
||
```{julia}
|
||
f(x) = x^3 - 2x - 5
|
||
x₀ = 2.0
|
||
x₁ = x₀ - f(x₀) / f'(x₀)
|
||
x₁, f(x₁)
|
||
```
|
||
|
||
We can see we are closer to a zero. Repeating, we have:
|
||
|
||
```{julia}
|
||
x₂ = x₁ - f(x₁)/ f'(x₁)
|
||
x₂, f(x₂)
|
||
```
|
||
|
||
And:
|
||
|
||
|
||
```{julia}
|
||
x₃ = x₂ - f(x₂)/ f'(x₂)
|
||
x₃, f(x₃)
|
||
```
|
||
|
||
```{julia}
|
||
x₄ = x₃ - f(x₃)/ f'(x₃)
|
||
x₄, f(x₄)
|
||
```
|
||
|
||
We see now that $f(x_4)$ is within machine tolerance of $0$ and that if we were to try another iteration we would find $x_{i+1} \approx x_i$. We call $x_4$ an *approximate zero* of $f(x)$.
|
||
|
||
::: {.definition title="Newton's method"}
|
||
|
||
Let $x_0$ be an initial guess for a zero of $f(x)$. Iteratively define $x_{i+1}$ in terms of $x_i$ by:
|
||
|
||
$$
|
||
x_{i+1} = x_i - \frac{f(x_i)}{f'(x_i)}.
|
||
$$
|
||
|
||
Then for reasonable functions and reasonable initial guesses, the sequence of points converges to a zero of $f$.
|
||
|
||
:::
|
||
|
||
|
||
On the computer, we know that actual convergence will likely never occur, but accuracy to a certain tolerance---either with $\lvert x_{i+1} - x_i\rvert$ or $\lvert f(x_i) \rvert$---can often be achieved.
|
||
|
||
|
||
In the example above, we tediously kept track of each value to match the formula for the update step of Newton's method. However, the subscripting mathematically is used to specify assignment, as opposed to an equation, and that is exactly what the equals sign does in `Julia`, so we could have just done these steps:
|
||
|
||
```{julia}
|
||
x = 2.0
|
||
x = x - f(x) / f'(x)
|
||
x = x - f(x) / f'(x)
|
||
x = x - f(x) / f'(x)
|
||
x = x - f(x) / f'(x)
|
||
```
|
||
|
||
In practice, the algorithm is implemented not by repeating the update step a fixed number of times, rather by repeating the step until either we "converge" or it is clear we won't converge. For good guesses and most functions, convergence happens quickly.
|
||
|
||
|
||
:::{.callout-note}
|
||
## Note
|
||
|
||
Newton looked at this same example in 1699 (B.T. Polyak, *Newton's method and its use in optimization*, European Journal of Operational Research. 02/2007; 181(3):1086-1096.; and Deuflhard *Newton Methods for Nonlinear Problems: Affine Invariance and Adaptive Algorithms*) though his technique was slightly different as he did not use the derivative, *per se*, but rather an approximation based on the fact that his function was a polynomial.
|
||
|
||
We can read that he guessed the answer was ``2 + p``, as there is a sign change between $2$ and $3$. Newton put this guess into the polynomial to get after simplification ``p^3 + 6p^2 + 10p - 1``. This has an **approximate** zero found by solving the linear part ``10p-1 = 0``. Taking ``p = 0.1`` he then can say the answer looks like ``2 + p + q`` and repeat to get ``q^3 + 6.3q^2 + 11.23q + 0.061 = 0``. Again taking just the linear part estimates `q = -0.005431...`. After two steps the estimate is `2.094568...`. This can be continued by expressing the answer as ``2 + p + q + r`` and then solving for an estimate for ``r``.
|
||
|
||
Raphson (1690) proposed a simplification avoiding the computation of new polynomials, hence the usual name of the Newton-Raphson method. Simpson introduced derivatives into the formulation and systems of equations.
|
||
|
||
:::
|
||
|
||
#### Examples
|
||
|
||
|
||
##### Example: visualizing convergence
|
||
|
||
|
||
@fig-newtons-method demonstrates the method and the rapid convergence:
|
||
|
||
::: {#fig-newtons-method}
|
||
```{julia}
|
||
#| echo: false
|
||
function newtons_method_graph(n, f, a, b, c; label=false)
|
||
|
||
xstars = [c]
|
||
xs = [c]
|
||
ys = [0.0]
|
||
|
||
plt = plot(f, a, b, legend=false, size=fig_size,
|
||
line = (:royalblue, 3),
|
||
axis = ([], false)
|
||
)
|
||
plot!(plt, [a, b], [0,0], color=:black)
|
||
|
||
|
||
ts = range(a, stop=b, length=50)
|
||
for i in 1:n
|
||
x0 = xs[end]
|
||
x1 = x0 - f(x0)/f'(x0)
|
||
push!(xstars, x1)
|
||
append!(xs, [x0, x1])
|
||
append!(ys, [f(x0), 0])
|
||
end
|
||
plot!(plt, xs, ys, color=:orange)
|
||
scatter!(plt, xstars, 0*xstars, color=:orange, markersize=5)
|
||
if label
|
||
annotate!(plt, [(xᵢ, 0, text(latexstring("x_$(i-1)"), :bottom,:left)) for (i, xᵢ) in enumerate(xstars)])
|
||
end
|
||
plt
|
||
end
|
||
|
||
let
|
||
gr()
|
||
caption = ""
|
||
n = 5
|
||
|
||
fn, a, b, c = x->log(x), .15, 2, .2
|
||
|
||
anim = @animate for i=1:n
|
||
newtons_method_graph(i-1, fn, a, b, c; label=true)
|
||
end
|
||
|
||
imgfile = tempname() * ".gif"
|
||
gif(anim, imgfile, fps = 1)
|
||
plotly()
|
||
ImageFile(imgfile, caption)
|
||
end
|
||
```
|
||
|
||
Illustration of Newton's Method converging to a zero of a function.
|
||
:::
|
||
|
||
|
||
##### Example non-polynomial
|
||
|
||
The first example by Newton of applying the method to a non-polynomial function was solving an equation from astronomy: $x - e \sin(x) = M$, where $e$ is an eccentric anomaly and $M$ a mean anomaly. Newton used polynomial approximations for the trigonometric functions, here we can solve directly.
|
||
|
||
Let $e = 1/2$ and $M = 3/4$. With $f(x) = x - e\sin(x) - M$ then $f'(x) = 1 - e \cos(x)$. Starting at 1, Newton's method for 3 steps becomes:
|
||
|
||
```{julia}
|
||
ec, M = 0.5, 0.75
|
||
f(x) = x - ec * sin(x) - M
|
||
fp(x) = 1 - ec * cos(x)
|
||
x = 1
|
||
x = x - f(x) / fp(x)
|
||
x = x - f(x) / fp(x)
|
||
x = x - f(x) / fp(x)
|
||
x, f(x)
|
||
```
|
||
|
||
##### Example: numeric not algebraic
|
||
|
||
|
||
For the function $f(x) = \cos(x) - x$ consider this SymPy code to symbolically solve for a zero:
|
||
|
||
```{julia}
|
||
#| error: true
|
||
#| eval: false
|
||
@syms x::real
|
||
solve(cos(x) ~ x, x)
|
||
```
|
||
|
||
Were this run it would produce an error
|
||
|
||
```
|
||
NotImplementedError('multiple generators [x, cos(x)]
|
||
No algorithms are implemented to solve equation -x + cos(x)')
|
||
```
|
||
|
||
Non-linear equations may not have exact symbolic answers. However,
|
||
With Newton's method we can readily find a numeric solution, even though there is no closed-form answer.
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
f(x) = cos(x) - x
|
||
x = 0.5
|
||
x = x - f(x)/f'(x) # 0.7552224171056364
|
||
x = x - f(x)/f'(x) # 0.7391416661498792
|
||
x = x - f(x)/f'(x) # 0.7390851339208068
|
||
x = x - f(x)/f'(x) # 0.7390851332151607
|
||
x = x - f(x)/f'(x)
|
||
x, f(x)
|
||
```
|
||
|
||
To machine tolerance the answer is a zero, even though the exact answer is irrational and all finite floating point values can be represented as rational numbers.
|
||
|
||
|
||
##### Example
|
||
|
||
|
||
Use Newton's method to find the *largest* real solution to $e^x = x^6$. A plot shows that that answer is *near* $x=20$, so we begin there. To use Newton's method to find an intersection point, we create a new function which is zero when the two functions are equal through subtraction.
|
||
|
||
For this problem we use a loop to illustrate the progression of the algorithm:
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| term: true
|
||
h(x) = exp(x) - x^6
|
||
x = 20
|
||
for step in 1:11
|
||
delta = h(x)/h'(x)
|
||
x = x - delta
|
||
@show step, x, delta
|
||
end
|
||
```
|
||
|
||
By the ninth step, the increment `delta`---which tracks $\lvert x_{i+1} - x_i \rvert$ is negligible and the algorithm has stopped improving. The approximate zero found is `16.99888735229605`.
|
||
|
||
##### Example division as multiplication
|
||
|
||
|
||
[Newton-Raphson Division](http://tinyurl.com/kjj9w92) is a means to divide by multiplying.
|
||
|
||
|
||
Why would you want to do that? Well, even for computers division is harder (read slower) than multiplying. The trick is that $p/q$ is simply $p \cdot (1/q)$, so finding a means to compute a reciprocal by multiplying will reduce division to multiplication.
|
||
|
||
|
||
Well suppose we have $q$, we could try to use Newton's method to find $1/q$, as it is a solution to $f(x) = x - 1/q$. The Newton update step simplifies to:
|
||
|
||
|
||
$$
|
||
x - f(x) / f'(x) \quad\text{or}\quad x - (x - 1/q)/ 1 = 1/q
|
||
$$
|
||
|
||
That doesn't really help, as Newton's method is just $x_{i+1} = 1/q$. That is, it just jumps to the answer, the one we want to compute by some other means!
|
||
|
||
|
||
Trying again, we simplify the update step for a related function: $f(x) = 1/x - q$ with $f'(x) = -1/x^2$ and then one step of the process is:
|
||
|
||
|
||
$$
|
||
x_{i+1} = x_i - (1/x_i - q)/(-1/x_i^2) = -qx^2_i + 2x_i.
|
||
$$
|
||
|
||
Now for $q$ in the interval $[1/2, 1]$ we want to get a *good* initial guess. Here is a claim: we can use $x_0=48/17 - 32/17 \cdot q$. We check graphically in @fig-plot-1-over-q-and-reasonable-initial-approximation that this is a reasonable initial approximation to $1/q$.
|
||
|
||
|
||
::: {#fig-plot-1-over-q-and-reasonable-initial-approximation}
|
||
```{julia}
|
||
#| hold: true
|
||
|
||
plot(q -> 1/q, 1/2, 1, label="1/q")
|
||
plot!(q -> 1/17 * (48 - 32q), label="linear approximation")
|
||
```
|
||
|
||
The linear approximation shows good starting point for Newton's method
|
||
:::
|
||
|
||
It can be shown that we have for any $q$ in $[1/2, 1]$ with initial guess $x_0 = 48/17 - 32/17\cdot q$ that Newton's method will converge to $16$ digits in no more than this many steps:
|
||
|
||
|
||
$$
|
||
\log_2(\frac{53 + 1}{\log_2(17)}).
|
||
$$
|
||
|
||
Computing, we see that four steps suffices.
|
||
|
||
```{julia}
|
||
a = log2((53 + 1)/log2(17))
|
||
ceil(Integer, a)
|
||
```
|
||
|
||
Now we try to find $1/q$ when $q=0.8 = 4/5$ without dividing by $q$.
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
q = 0.80
|
||
x = (48/17) - (32/17)*q
|
||
x = -q*x*x + 2*x
|
||
x = -q*x*x + 2*x
|
||
x = -q*x*x + 2*x
|
||
x = -q*x*x + 2*x
|
||
```
|
||
|
||
If values for `48/17` and `32/17` are pre-computed, this method has basically $18$ multiplication and addition operations for one division, so it naively would seem slower, but timing this shows the method is competitive with a regular division.
|
||
|
||
|
||
## Automating Newton's method
|
||
|
||
In the previous examples, we saw fast convergence, guaranteed converge in $4$ steps, and an example where $9$ steps were needed to get convergence. Newton's method usually converges quickly, but may converge slowly, and may not converge at all. Automating the task to avoid repeatedly running the update step is a task best done by the computer.
|
||
|
||
|
||
The `while` loop is a good way to repeat commands until some condition is met. With this, we present a simple function implementing Newton's method, we iterate until the update step gets really small (the `atol`) or the convergence takes more than $50$ steps. (There are other, better choices that could be used to determine when the algorithm should stop, these are just easy to understand.)
|
||
|
||
|
||
```{julia}
|
||
function nm(f, fp, x0)
|
||
atol = 1e-14
|
||
ctr = 0
|
||
delta = Inf
|
||
while (abs(delta) > atol) && (ctr < 50)
|
||
delta = f(x0)/fp(x0)
|
||
x0 = x0 - delta
|
||
ctr = ctr + 1
|
||
end
|
||
|
||
ctr < 50 ? x0 : NaN
|
||
end
|
||
```
|
||
|
||
##### Examples
|
||
|
||
|
||
* Find a zero of $\sin(x)$ starting at $x_0=3$:
|
||
|
||
|
||
```{julia}
|
||
nm(sin, cos, 3)
|
||
```
|
||
|
||
This is an approximation for $\pi$, that historically found use, as the convergence is fast.
|
||
|
||
|
||
* Find a solution to $x^5 = 5^x$ near $2$:
|
||
|
||
|
||
Writing a function to handle this, we have:
|
||
|
||
|
||
```{julia}
|
||
k(x) = x^5 - 5^x
|
||
```
|
||
|
||
We could find the derivative by hand, but use the automatic one instead:
|
||
|
||
|
||
```{julia}
|
||
alpha = nm(k, k', 2)
|
||
alpha, k(alpha)
|
||
```
|
||
|
||
### `Roots.Newton()`
|
||
|
||
|
||
Typing in the `nm` function might be okay once, but would be tedious if it was needed each time. Besides, it isn't as robust to different inputs as possible. The `Roots` package provides a `Newton` method for its `find_zero` function.
|
||
|
||
|
||
To use a different method with `find_zero`, the calling pattern is `find_zero(f, x, M)` where `f` represent the function(s), `x` the initial point(s), and `M` the method. For `Newton` we have:
|
||
|
||
|
||
```{julia}
|
||
find_zero((sin, cos), 3, Roots.Newton())
|
||
```
|
||
|
||
Or, if a derivative is not specified, one can be computed using automatic differentiation:
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
f(x) = sin(x)
|
||
find_zero((f, f'), 2, Roots.Newton())
|
||
```
|
||
|
||
The `Newton` method isn't exported, so it is qualified via `Roots.Newton()`.
|
||
|
||
|
||
##### Example: solving $f(x) = c$ for non-zero $c$
|
||
|
||
Find a value for which `erf(x) = 0.75`.
|
||
|
||
The `erf` function is increasing, so there is just one value and an exploratory graph shows the answer to be near $1$.
|
||
|
||
We can apply Newton's method, but first we need to restate the problem in terms of some function equaling $0$. This can be done directly, but we do it in two steps here:
|
||
|
||
```{julia}
|
||
f(x) = erf(x)
|
||
c = 0.75
|
||
h(x) = f(x) - c
|
||
find_zero((h, h'), 1.0, Roots.Newton())
|
||
```
|
||
|
||
##### Example: intersection of two graphs, or solving $f(x) = g(x)$
|
||
|
||
|
||
Find the intersection point between $f(x) = \cos(x)$ and $g(x) = 5x$ near $0$.
|
||
|
||
|
||
We have Newton's method to solve for zeros of $f(x)$, i.e. when $f(x) = 0$. Here we want to solve for $x$ with $f(x) = g(x)$. To do so, we make a new function $h(x) = f(x) - g(x)$, that is $0$ when $f(x)$ equals $g(x)$:
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
f(x) = cos(x)
|
||
g(x) = 5x
|
||
h(x) = f(x) - g(x)
|
||
x0 = find_zero((h, h'), 0, Roots.Newton())
|
||
x0, h(x0), f(x0) - g(x0)
|
||
```
|
||
|
||
---
|
||
|
||
|
||
We redo the above using a *parameter* for the $5$, as there are some options on how it would be done. We let `f(x,p) = cos(x) - p*x`. Then we can use `Roots.Newton` by also defining a derivative:
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
f(x,p) = cos(x) - p*x
|
||
fp(x,p) = -sin(x) - p
|
||
xn = find_zero((f,fp), pi/4, Roots.Newton(); p=5)
|
||
xn, f(xn, 5)
|
||
```
|
||
|
||
To use automatic differentiation with a parameter is not straightforward, as we must hold the `p` fixed. For this, we introduce a closure that fixes `p` and differentiates in the `x` variable (called `u` below):
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
f(x,p) = cos(x) - p*x
|
||
fp(x,p) = (u -> f(u,p))'(x)
|
||
xn = find_zero((f,fp), pi/4, Roots.Newton(); p=5)
|
||
```
|
||
|
||
##### Example: finding $c$ in Rolle's Theorem
|
||
|
||
|
||
The function $r(x) = \sqrt{1 - \cos(x^2)^2}$ has a zero at $0$ and one at $a$ near $1.77$, as can be seen in @fig-sqrt-1-cos-x-squared-squared.
|
||
|
||
::: {#fig-sqrt-1-cos-x-squared-squared}
|
||
|
||
```{julia}
|
||
#| echo: false
|
||
r(x) = sqrt(1 - cos(x^2)^2)
|
||
tks = 0:0.5:1.5
|
||
plot(r, 0, 1.77, xticks = (vcat(tks, 1.77), vcat(string.(tks), L"a")))
|
||
```
|
||
|
||
Plot of $r(x) = \sqrt{1 - \cos(x^2)^2}$ over $[0, 1.77]$
|
||
:::
|
||
|
||
|
||
As $f(x)$ is differentiable between $0$ and $a$, Rolle's theorem says there will be value where the derivative is $0$. Find that value.
|
||
|
||
|
||
This value will be a zero of the derivative. @fig-sqrt-1-cos-x-squared-squared shows it should be near $1.2$, so we use that as a starting value to get the answer:
|
||
|
||
|
||
```{julia}
|
||
find_zero((r',r''), 1.2, Roots.Newton())
|
||
```
|
||
|
||
##### Example: seeing the trace
|
||
|
||
The steps of Newton's method can be see by passing a `Roots.Tracks` object. We name it `tracks` in this example to take advantage of `Julia`'s handling of matching variables with keywords with the same name ([argument destructuring](https://docs.julialang.org/en/v1/manual/functions/#man-argument-destructuring)).
|
||
|
||
Consider finding a zero of $f(x) = e^{x} - x^{\pi}$. There is one near $2$.
|
||
|
||
We have two additional steps to see the trace, first we create a `tracks` object.
|
||
|
||
```{julia}
|
||
f(x) = exp(x) - x^pi
|
||
x0 = 2
|
||
tracks = Roots.Tracks()
|
||
find_zero((f, f'), x0, Roots.Newton(); tracks)
|
||
```
|
||
|
||
The we display the `tracks` object to see the steps taken by the algorithm along with some diagnostic details:
|
||
|
||
```{julia}
|
||
tracks
|
||
```
|
||
|
||
## The secant method
|
||
|
||
The secant method is an alternative to Newton's method which uses secant lines instead of tangent lines in the update step. Like Newton's method, the secant method is iterative. Unlike Newton's method---which uses just the previous value to identify the next value---the secant method uses the two previous values in its update step.
|
||
|
||
::: {.relationship title="Secant method"}
|
||
|
||
Let $x_0$ and $x_1$ be two different estimates for $c$, a zero of $f(x)$. The iterative algorithm with update step
|
||
|
||
$$
|
||
x_{i+1} = x_i - \frac{x_{i} - x_{i-1}}{f(x_{i}) - f(x_{i-1})} \cdot f(x_i)
|
||
$$
|
||
|
||
is called the secant method.
|
||
:::
|
||
|
||
The multiplier of $f(x_i)$ is the reciprocal of the slope of the secant line from $(x_i, f(x_i))$ and $(x_{i-1}, f(x_{i-1}))$---in contrast to Newton's method which uses the reciprocal of the slope of the tangent line at $(x_i, f(x_i))$.
|
||
|
||
The secant method can be preferred if either a function's evaluation or a function's derivative evaluation are difficult to find.
|
||
|
||
##### Example
|
||
|
||
Find a zero of $f(x) = \cos(x) - x$ using the secant method starting from $x_0, x_1 = 0, \pi/2$.
|
||
|
||
```{julia}
|
||
f(x) = cos(x) - x
|
||
xi_1, xi = 0, pi/2
|
||
xi, xi_1 = xi - (xi - xi_1) / (f(xi) - f(xi_1)) * f(xi), xi # x2 = 0.6110154703516573
|
||
xi, xi_1 = xi - (xi - xi_1) / (f(xi) - f(xi_1)) * f(xi), xi # x3 = 0.7232695414357495
|
||
xi, xi_1 = xi - (xi - xi_1) / (f(xi) - f(xi_1)) * f(xi), xi # x4 = 0.739567106974727
|
||
xi, xi_1 = xi - (xi - xi_1) / (f(xi) - f(xi_1)) * f(xi), xi # x5 = 0.7390834365030763
|
||
xi, xi_1 = xi - (xi - xi_1) / (f(xi) - f(xi_1)) * f(xi), xi # x6 = 0.739085133034638
|
||
xi, xi_1 = xi - (xi - xi_1) / (f(xi) - f(xi_1)) * f(xi), xi # x7 = 0.7390851332151608
|
||
```
|
||
|
||
This example takes six iterations to reach convergence to machine tolerance. Newton's method takes only five, as in general it converges more rapidly. However, the secant method takes just one function evaluation per step, or $6$ in total not counting the initial two values; Newton's method takes $2$ per step or $10$ in total, not counting the initial value. In general this holds: the secant method might take more steps, but will require fewer function calls.
|
||
|
||
<!-- https://www.math.kent.edu/~reichel/courses/optimization/reading.material.1/secant.pdf -->
|
||
::: {.callout-note}
|
||
## Origin of secant method
|
||
|
||
[Papakonstantinou and Tapia](https://www.jstor.org/stable/10.4169/amer.math.monthly.120.06.500?origin=JSTOR-pdf) discuss the origin of the secant method tracing it back to the rule of double false position used in some manner since the 18th century BC which in modern language translates to one step of the secant method and when applied to a linear function gives a solution.
|
||
|
||
:::
|
||
|
||
### `Roots.Secant()`
|
||
|
||
The `Roots` package has a `Secant` method for `find_zero` to carry out this method. We redo the last example, using a `Roots.tracks` object so we can see the algorithm, though in most cases this is not of interest.
|
||
|
||
|
||
```{julia}
|
||
f(x) = cos(x) - x
|
||
xs = (0, pi/2)
|
||
tracks = Roots.Tracks()
|
||
find_zero(f, xs, Secant(); tracks)
|
||
```
|
||
|
||
The tracks are identical up to floating point differences to those in the comments in the previous example.
|
||
|
||
```{julia}
|
||
tracks
|
||
```
|
||
|
||
|
||
#### Default method for a nearby initial guess
|
||
|
||
The `find_zero` function has a default, `Roots.Order0()`, when a single *nearby* value is specified as a starting point. This method use a secant method (after using an approximate derivative to get the second step) up *until* convergence *or* the values $x_{i-1}$ and $x_i$ form a bracketing interval. If the latter happens, then a bracketing method is used to find convergence. (Bracketing methods have guaranteed convergence).
|
||
|
||
For example, we might have:
|
||
|
||
```{juiia}
|
||
empty!(tracks) # can empty tracks or create a new one
|
||
x0 = first(xs) # just a single value, not even a good initial guess here
|
||
find_zero(f, x0; tracks) # no method specified so defaults to Order0()
|
||
```
|
||
|
||
The trace shows the steps of the secant method until a bracket is identified and then the brackets up to convergence.
|
||
|
||
```{julia}
|
||
tracks
|
||
```
|
||
|
||
The `Order0` method isn't quite as convergent as Newton's method, but is a bit more robust to some of that methods idiosyncrasies that are discussed later in this section.
|
||
|
||
## Convergence rates
|
||
|
||
|
||
Newton's method is famously known to have "quadratic convergence". What does this mean?
|
||
|
||
When it works, Newton's method forms a sequence $x_0, x_1, x_2, \dots$, converging to a zero, $\alpha$ of some function $f(x)$.
|
||
|
||
Define error in the $i$th step by:
|
||
|
||
$$
|
||
e_i = x_i - \alpha.
|
||
$$
|
||
|
||
We take the *order of convergence* to be the value $p$ for which
|
||
|
||
|
||
$$
|
||
\lim_{n \rightarrow \infty} \frac{e_{n+1}}{e_n^p} = L > 0.
|
||
$$
|
||
|
||
We say, the sequence converges with order $p$. Quadratic convergence is when $p=2$.
|
||
|
||
|
||
|
||
### Convergence of Newton's method
|
||
|
||
We will see Newton's method satisfies a bound like this:
|
||
|
||
|
||
$$
|
||
\lvert e_{i+1} \rvert \leq M_i \cdot e_i^2.
|
||
$$
|
||
|
||
In fact, we will see that under assumptions the value for $M$ will converge to $f''(\alpha)/(2f'(\alpha))$.
|
||
|
||
If $M$ were just a constant in the above and we suppose a good initial guess, say with $e_0 = 10^{-1}$, then $e_1$ would be less than $M 10^{-2}$ and $e_2$ less than $M^2 10^{-4}$, $e_3$ less than $M^3 10^{-8}$ and $e_4$ less than $M^4 10^{-16}$ which for $M=1$ is basically the machine precision when values are near $1$. That is for some problems, with a good initial guess it will take around $4$ or so steps to converge.
|
||
|
||
|
||
To identify $M$, *assume*
|
||
|
||
* The function $f$ has a continuous *second* derivative in a neighborhood of $\alpha$.
|
||
|
||
* The value $f'(\alpha)$ is *non-zero* in the neighborhood of $\alpha$.^[This property says that this is a *simple* zero or the zero has *multiplicity* of $1$.]
|
||
|
||
|
||
Then the Lagrange remainder form for linearization holds at each $x_i$ in the above neighborhood:
|
||
|
||
|
||
$$
|
||
f(x) = f(x_i) + f'(x_i) \cdot (x - x_i) + \frac{1}{2} f''(\xi) \cdot (x-x_i)^2.
|
||
$$
|
||
|
||
The value $\xi$ is from the mean value theorem and is between $x$ and $x_i$.
|
||
|
||
|
||
Setting $x=\alpha$ (as $f(\alpha)=0$) and dividing by $f'(x_i)$ leaves:
|
||
|
||
|
||
$$
|
||
0 = \frac{f(\alpha)}{f'(x_i)} = \frac{f(x_i)}{f'(x_i)} + (\alpha-x_i) + \frac{1}{2}\cdot \frac{f''(\xi)}{f'(x_i)} \cdot (\alpha-x_i)^2.
|
||
$$
|
||
|
||
We can write $e_{i+1}$ in terms of $x_i$ and the update step and simplify using the above relationship:
|
||
|
||
|
||
$$
|
||
\begin{align*}
|
||
x_{i+1} - \alpha
|
||
&= \left(x_i - \frac{f(x_i)}{f'(x_i)}\right) - \alpha\\
|
||
&= \left(x_i - \alpha \right) - \frac{f(x_i)}{f'(x_i)}\\
|
||
&= (x_i - \alpha) + \left(
|
||
(\alpha - x_i) + \frac{1}{2}\frac{f''(\xi) \cdot(\alpha - x_i)^2}{f'(x_i)}
|
||
\right)\\
|
||
&= \frac{1}{2}\frac{f''(\xi)}{f'(x_i)} \cdot(x_i - \alpha)^2.
|
||
\end{align*}
|
||
$$
|
||
|
||
|
||
That is, $M$ can be read off from this equality:
|
||
|
||
|
||
$$
|
||
e_{i+1} = \frac{1}{2}\frac{f''(\xi)}{f'(x_i)} e_i^2.
|
||
$$
|
||
|
||
This convergence to $\alpha$ will be quadratic *if*:
|
||
|
||
|
||
* The initial guess $x_0$ is *near* $\alpha$, so $e_0$ is managed.
|
||
|
||
* The derivative at $\alpha$ is not too close to $0$, hence, by continuity $f'(x_i)$ is not too close to $0$. (As it appears in the denominator). That is, the function can't be too flat, which should make sense, as then the tangent line is nearly parallel to the $x$ axis and would intersect far away or the algorithm can get trapped by a local extrema.
|
||
|
||
* The function $f$ has a continuous second derivative at $\alpha$.
|
||
|
||
* The second derivative is not too big (in absolute value) near $\alpha$. A large second derivative means the function is very concave, which means it is "turning" a lot. In this case, the function turns away from the tangent line quickly, so the tangent line's zero is not necessarily a good approximation to the actual zero, $\alpha$.
|
||
|
||
|
||
:::{.callout-note}
|
||
## Tradeoffs
|
||
|
||
The bisection method has linear convergence, in that $\lvert e_{i+1} \rvert \approx (1/2) \lvert e_i \rvert$, but *guaranteed* to converge.
|
||
|
||
Newton's method is quadratic, so *can* converge in a few steps---but convergence is not guaranteed.
|
||
|
||
:::
|
||
|
||
|
||
|
||
|
||
### When Newton's method fails
|
||
|
||
What can go wrong when one of these isn't the case is illustrated next:
|
||
|
||
|
||
#### Poor initial guess
|
||
|
||
::: {#fig-poor-initial-step-newtons-method}
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
#| cache: true
|
||
### {{{newtons_method_poor_x0}}}
|
||
gr()
|
||
caption = ""
|
||
|
||
fn, a, b, c = x -> sin(x) - x/4, -15, 20, 2pi
|
||
|
||
n = 20
|
||
anim = @animate for i=1:n
|
||
newtons_method_graph(i-1, fn, a, b, c)
|
||
end
|
||
|
||
imgfile = tempname() * ".gif"
|
||
gif(anim, imgfile, fps = 2)
|
||
plotly()
|
||
ImageFile(imgfile, caption)
|
||
```
|
||
|
||
|
||
Illustration of Newton's Method converging to a zero of a function,
|
||
but slowly as the initial guess, is very poor, and not close to the
|
||
zero. The algorithm does converge in this illustration, but not quickly and not to the nearest root from
|
||
the initial guess.
|
||
|
||
:::
|
||
|
||
::: {#fig-poor-initial-step-newtons-method-example-2}
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
#| cache: true
|
||
# {{{newtons_method_flat}}}
|
||
gr()
|
||
caption = ""
|
||
|
||
fn, a, b, c = x -> x^5 - x + 1, -1.5, 1.4, 0.0
|
||
|
||
n=7
|
||
anim = @animate for i=1:n
|
||
newtons_method_graph(i-1, fn, a, b, c)
|
||
end
|
||
imgfile = tempname() * ".gif"
|
||
gif(anim, imgfile, fps = 1)
|
||
plotly()
|
||
ImageFile(imgfile, caption)
|
||
```
|
||
|
||
Illustration of Newton's method failing to converge as for some $x_i$,
|
||
$f'(x_i)$ is too close to $0$. In this instance after a few steps, the
|
||
algorithm just cycles around the local minimum near $0.66$. The values
|
||
of $x_i$ repeat in the pattern: $1.0002, 0.7503, -0.0833, 1.0002,
|
||
\dots$. This is also an illustration of a poor initial guess. If there
|
||
is a local minimum or maximum between the guess and the zero, such
|
||
cycles can occur.
|
||
|
||
:::
|
||
|
||
#### The second derivative is too big
|
||
|
||
::: {#fig-second-derivative-too-big-newtons-method}
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
#| cache: true
|
||
# {{{newtons_method_cycle}}}
|
||
gr()
|
||
fn, a, b, c, = x -> abs(x)^(0.49), -2, 2, 1.0
|
||
caption = ""
|
||
|
||
n=10
|
||
anim = @animate for i=1:n
|
||
newtons_method_graph(i-1, fn, a, b, c)
|
||
end
|
||
|
||
imgfile = tempname() * ".gif"
|
||
gif(anim, imgfile, fps = 2)
|
||
|
||
plotly()
|
||
ImageFile(imgfile, caption)
|
||
```
|
||
|
||
Illustration of Newton's Method not converging. Here the second
|
||
derivative is too big near the zero - it blows up near $0$ - and the
|
||
convergence does not occur. Rather the iterates increase in their
|
||
distance from the zero.
|
||
|
||
:::
|
||
|
||
|
||
#### The tangent line at some xᵢ is flat
|
||
|
||
::: {#fig-tangent-line-too-flat-at-x0}
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
#| cache: true
|
||
# {{{newtons_method_wilkinson}}}
|
||
gr()
|
||
caption = ""
|
||
|
||
fn,a,b,c = x -> x^20 - 1, .7, 1.4, 8/9
|
||
n = 10
|
||
|
||
anim = @animate for i=1:n
|
||
newtons_method_graph(i-1, fn, a, b, c)
|
||
end
|
||
imgfile = tempname() * ".gif"
|
||
gif(anim, imgfile, fps = 1)
|
||
plotly()
|
||
ImageFile(imgfile, caption)
|
||
```
|
||
|
||
The function $f(x) = x^{20} - 1$ has two bad behaviours for Newton's
|
||
method: for $x < 1$ the derivative is nearly $0$ and for $x>1$ the
|
||
second derivative is very big. In this illustration, we have an
|
||
initial guess of $x_0=8/9$. As the tangent line is fairly flat, the
|
||
next approximation is far away, $x_1 = 1.313\dots$. As this guess
|
||
is much bigger than $1$, the ratio $f(x)/f'(x) \approx
|
||
x^{20}/(20x^{19}) = x/20$, so $x_i - f(x_i)/f'(x_i) \approx (19/20)x_i$
|
||
yielding slow, linear convergence until $f''(x_i)$ is moderate. For
|
||
this function, starting at $x_0=8/9$ takes 11 steps, at $x_0=7/8$
|
||
takes 13 steps, at $x_0=3/4$ takes ``55`` steps, and at $x_0=1/2$ it takes
|
||
$204$ steps.
|
||
|
||
:::
|
||
|
||
|
||
###### Example: roots with multiplicity more than one
|
||
|
||
|
||
The assumption that $f'(\alpha)$ is non zero says $\alpha$ is a simple zero for $f(x)$. Near enough around $\alpha$, quadratic convergence should apply. However, consider the function $g(x) = f(x)^k$ for some integer $k \geq 2$. Then $\alpha$ is still a zero, but the derivative of $g$ at $\alpha$ is zero, so the tangent line is basically flat. This will slow the convergence up. We can see that the update step $g(x)/g'(x)$ becomes $(1/k) f(x)/f'(x)$, so an extra factor is introduced.
|
||
|
||
|
||
The calculation that produces the quadratic convergence now becomes:
|
||
|
||
|
||
$$
|
||
\begin{align*}
|
||
x_{i+1} - \alpha &= (x_i - \alpha) - \frac{1}{k}(x_i-\alpha - \frac{f''(\xi)}{2f'(x_i)}(x_i-\alpha)^2) \\
|
||
&= \frac{k-1}{k} (x_i-\alpha) + \frac{f''(\xi)}{2kf'(x_i)}(x_i-\alpha)^2.
|
||
\end{align*}
|
||
$$
|
||
|
||
As $k > 1$, the $(x_i - \alpha)$ term dominates, and we see the convergence is linear with $\lvert e_{i+1}\rvert \approx \left((k-1)/k\right) \lvert e_i\rvert$.
|
||
|
||
### Convergence of the secant method
|
||
|
||
As above, let $\epsilon_{n+1} = x_{n+1}-\alpha$ and *assume* $f'(\alpha) \neq 0$, or $\alpha$ is a *simple* zero of $f(x)$.
|
||
|
||
|
||
With a more involved derivation than that for Newton's method, a [calculation](https://math.okstate.edu/people/binegar/4513-F98/4513-l08.pdf) shows that
|
||
|
||
|
||
$$
|
||
\begin{align*}
|
||
\epsilon_{n+1}
|
||
& \approx \frac{f''(\alpha)}{2f'(\alpha)} \epsilon_n \epsilon_{n-1}\\
|
||
&= C \epsilon_n \epsilon_{n-1}.
|
||
\end{align*}
|
||
$$
|
||
|
||
|
||
The constant `C` is similar to that for Newton's method, and reveals potential troubles for the secant method similar to those of Newton's method: a poor initial guess (the initial error is too big), the second derivative is too large, the first derivative too flat near the answer.
|
||
|
||
|
||
Assuming the error term has the form $\lvert \epsilon_{n+1}\rvert = A|\epsilon_n|^\phi$ and substituting into the above leads to the equation
|
||
|
||
|
||
$$
|
||
\frac{A^{1+1/\phi}}{C} = |\epsilon_n|^{1 - \phi +1/\phi}.
|
||
$$
|
||
|
||
The left side being a constant suggests $\phi$ solves: $1 - \phi + 1/\phi = 0$ or $\phi^2 -\phi - 1 = 0$. The solution is the golden ratio, $(1 + \sqrt{5})/2 \approx 1.618\dots$. That is convergence is super linear, but not quadratic, as Newton's method is.
|
||
|
||
|
||
## Questions
|
||
|
||
|
||
###### Question
|
||
|
||
|
||
@fig-graph-of-airyai-minus-3.3-to-0-x0-minus-2-point-8 shows a graph of some $f(x)$ with $x_0$ marked with a point:
|
||
|
||
::: {#fig-graph-of-airyai-minus-3.3-to-0-x0-minus-2-point-8}
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
import SpecialFunctions: airyai
|
||
let
|
||
gr()
|
||
p = plot(airyai, -3.3, 0; legend=false);
|
||
plot!(p, zero, -3.3, 0);
|
||
scatter!(p, [(-2.8, 0)], marker=(:orange, 5));
|
||
annotate!(p, [(-2.8, 0.0, text(L"x_0", :top))])
|
||
plotly()
|
||
p
|
||
end
|
||
```
|
||
|
||
Plot of $f(x)$ with a zero and an initial starting point for Newton's method marked with $x_0$.
|
||
:::
|
||
|
||
If one step of Newton's method was used, what would be the value of $x_1$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
choices = ["``-2.224``", "``-2.80``", "``-0.020``", "``0.355``"]
|
||
answ = 1
|
||
radioq(choices, answ, keep_order=true)
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
@fig-plot-some-increasing-concave-up-function show a graph of some increasing, concave up $f(x)$ with initial point $x_0$ marked. Let $\alpha$ be the zero.
|
||
|
||
::: {#fig-plot-some-increasing-concave-up-function}
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
p = plot(x -> x^2 - 2, .75, 2.2, legend=false);
|
||
plot!(p, zero, color=:green);
|
||
scatter!(p, [(1,0)], color=:orange, markersize=5);
|
||
annotate!(p, [(1,0, text(L"x_0",:top, :left)),
|
||
(sqrt(2), 0, text(L"\alpha", :top))]);
|
||
p
|
||
```
|
||
|
||
Graph of an increasing, concave up function.
|
||
:::
|
||
|
||
What can be said about $x_1$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
choices = [
|
||
L"It must be $x_1 > \alpha$",
|
||
L"It must be $x_1 < x_0$",
|
||
L"It must be $x_0 < x_1 < \alpha$"
|
||
]
|
||
answ = 1
|
||
radioq(choices, answ)
|
||
```
|
||
|
||
---
|
||
|
||
|
||
@fig-plot-some-increasing-concave-up-function-start-on-right is a graph of some increasing, concave up $f(x)$ with initial point $x_0$ marked. Let $\alpha$ be the zero.
|
||
|
||
::: {#fig-plot-some-increasing-concave-up-function-start-on-right}
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
let
|
||
gr()
|
||
plt = plot(x -> x^2 - 2, .75, 2.2; empty_style...)
|
||
plot!(plt, zero; line=(:green,), arrow=true, side=:right)
|
||
scatter!(plt, [2],[0], marker = (:orange, 5))
|
||
annotate!(plt, [
|
||
(2, 0, text(L"x_0",:top, :left)),
|
||
(sqrt(2), 0, text(L"\alpha", :top))])
|
||
plotly()
|
||
plt
|
||
end
|
||
```
|
||
|
||
Graph of an increasing, concave up function.
|
||
:::
|
||
What can be said about $x_1$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
choices = [
|
||
L"It must be $x_1 < \alpha$",
|
||
L"It must be $x_1 > x_0$",
|
||
L"It must be $\alpha < x_1 < x_0$"
|
||
]
|
||
answ = 3
|
||
radioq(choices, answ)
|
||
```
|
||
|
||
---
|
||
|
||
|
||
Suppose $f(x)$ is increasing and concave up. From the tangent line representation: $f(x) = f(c) + f'(c)\cdot(x-c) + f''(\xi)/2 \cdot(x-c)^2$, explain why it must be that the graph of $f(x)$ lies on or *above* the tangent line.
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
choices = [
|
||
L"As $f''(\xi)/2 \cdot(x-c)^2$ is non-negative, we must have $f(x) - (f(c) + f'(c)\cdot(x-c)) \geq 0$.",
|
||
L"As $f''(\xi) < 0$ it must be that $f(x) - (f(c) + f'(c)\cdot(x-c)) \geq 0$.",
|
||
L"This isn't true. The function $f(x) = x^3$ at $x=0$ provides a counterexample"
|
||
]
|
||
answ = 1
|
||
radioq(choices, answ)
|
||
```
|
||
|
||
This question can be used to give a proof for the previous two questions, which can be answered by considering the graphs alone. Combined, they say that if a function is increasing and concave up and $\alpha$ is a zero, then if $x_0 < \alpha$ it will be $x_1 > \alpha$, and for any $x_i > \alpha$, $\alpha \le x_{i+1} \le x_i$, so the sequence in Newton's method is decreasing and bounded below; conditions for which it is guaranteed mathematically there will be convergence.
|
||
|
||
|
||
###### Question
|
||
|
||
|
||
Let $f(x) = x^2 - 3^x$. This has derivative $2x - 3^x \cdot \log(3)$. Starting with $x_0=0$, what does Newton's method converge on?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
f(x) = x^2 - 3^x;
|
||
fp(x) = 2x - 3^x*log(3);
|
||
val = Roots.newton(f, fp, 0);
|
||
numericq(val, 1e-1)
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
Let $f(x) = \exp(x) - x^4$. There are 3 zeros for this function. Which one does Newton's method converge to when $x_0=2$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
f(x) = exp(x) - x^4;
|
||
fp(x) = exp(x) - 4x^3;
|
||
xstar= Roots.newton(f, fp, 2);
|
||
numericq(xstar, 1e-1)
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
Let $f(x) = \exp(x) - x^4$. As mentioned, there are 3 zeros for this function. Which one does Newton's method converge to when $x_0=8$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
f(x) = exp(x) - x^4;
|
||
fp(x) = exp(x) - 4x^3;
|
||
xstar = Roots.newton(f, fp, 8);
|
||
numericq(xstar, 1e-1)
|
||
```
|
||
|
||
###### Question
|
||
|
||
Let $f(x) = \exp(x) - x^4$. As mentioned, there are 3 zeros for this function. What does the *secant* method converge to when started with $x_0 = 0, x_1 = 3$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
f(x) = exp(x) - x^4;
|
||
xs = (0, 3)
|
||
xstar = find_zero(f, xs, Secant())
|
||
numericq(xstar, 1e-1)
|
||
```
|
||
|
||
###### Question
|
||
|
||
Let $f(x) = \exp(x) - x^4$. As mentioned, there are 3 zeros for this function. What does the *secant* method converge to when started with $x_0 = 3, x_1 = 6$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
f(x) = exp(x) - x^4;
|
||
xs = (3, 6)
|
||
xstar = find_zero(f, xs, Secant())
|
||
numericq(xstar, 1e-1)
|
||
```
|
||
|
||
###### Question
|
||
|
||
Let $f(x) = \exp(x) - x^4$. As mentioned, there are 3 zeros for this function. What does the *secant* method converge to when started with $x_0 = 6, x_1 = 9$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
f(x) = exp(x) - x^4;
|
||
xs = (6, 9)
|
||
xstar = find_zero(f, xs, Secant())
|
||
numericq(xstar, 1e-1)
|
||
```
|
||
|
||
|
||
###### Question
|
||
|
||
|
||
Let $f(x) = \sin(x) - \cos(4\cdot x)$.
|
||
|
||
|
||
Starting at $\pi/8$, solve for the root returned by Newton's method.
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
k1=4
|
||
f(x) = sin(x) - cos(k1*x);
|
||
fp(x) = cos(x) + k1*sin(k1*x);
|
||
val = Roots.newton(f, fp, pi/(2k1));
|
||
numericq(val)
|
||
```
|
||
|
||
|
||
###### Question
|
||
|
||
|
||
Let $f(x) = \sin(x) - \cos(4\cdot x)$.
|
||
|
||
|
||
Starting at $x_0 = 0, x_1 = 1$, solve for the root returned by secant method.
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
k1=4
|
||
f(x) = sin(x) - cos(k1*x)
|
||
xs = (0, 1)
|
||
val = find_zero(f, xs, Secant())
|
||
numericq(val)
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
Using Newton's method find a root to $f(x) = \cos(x) - x^3$ starting at $x_0 = 1/2$.
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
f(x) = cos(x) - x^3
|
||
val = Roots.newton(f,f', 1/2)
|
||
numericq(val)
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
Use Newton's method to find a root of $f(x) = x^5 + x -1$. Make a quick graph to find a reasonable starting point.
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
f(x) = x^5 + x - 1
|
||
val = Roots.newton(f,f', -1)
|
||
numericq(val)
|
||
```
|
||
###### Question
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
##Consider the following illustration of Newton's method:
|
||
caption = """
|
||
Illustration of Newton's method. Moving the point ``x_0`` shows different behaviours of the algorithm.
|
||
"""
|
||
## JSXGraph(:derivatives, "newtons-method.js", caption)
|
||
nothing
|
||
```
|
||
|
||
For the following graph, graphically consider the algorithm for a few different starting points.
|
||
|
||
::: {#fig-newton-method-consider-x0}
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
# placeholder until CWJ bumps up a version?
|
||
plot(x -> x^5 - x - 1, -1, 1.5; legend=false, line=(:black,))
|
||
plot!(zero; line=(:black,))
|
||
```
|
||
|
||
Plot of $f(x)$. Consider Newton's method for different initial values.
|
||
:::
|
||
|
||
|
||
If $x_0$ is $1$ what occurs?
|
||
|
||
|
||
```{julia}
|
||
#| echo: false
|
||
nm_choices = [
|
||
"The algorithm converges very quickly. A good initial point was chosen.",
|
||
"The algorithm converges, but slowly. The initial point is close enough to the answer to ensure decreasing errors.",
|
||
"The algorithm fails to converge, as it cycles about"
|
||
]
|
||
radioq(nm_choices, 1, keep_order=true)
|
||
```
|
||
|
||
When $x_0 = 1.0$ the following values are true for $f$:
|
||
|
||
|
||
```{julia}
|
||
#| echo: false
|
||
ff(x) = x^5 - x - 1
|
||
α = find_zero(ff, 1)
|
||
function error_terms(x)
|
||
(e₀=x-α, f₀′= ff'(x), f̄₀′′=ff''(α), ē₁ = 1/2*ff''(α)/ff'(x)*(x-α)^2)
|
||
end
|
||
error_terms(1.0)
|
||
```
|
||
|
||
Where the values `f̄₀′′` and `ē₁` are worst-case estimates when $\xi$ is between $x_0$ and the zero.
|
||
|
||
|
||
Does the magnitude of the error increase or decrease in the first step?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
radioq(["Appears to increase", "It decreases"],2,keep_order=true)
|
||
```
|
||
|
||
If $x_0$ is set near $0.50$ what happens?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
radioq(nm_choices, 3, keep_order=true)
|
||
```
|
||
|
||
When $x_0 = 0.5$ the following values are true for $f$:
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
error_terms(0.5)
|
||
```
|
||
|
||
Where the values `f̄₀′′` and `ē₁` are worst-case estimates when $\xi$ is between $x_0$ and the zero.
|
||
|
||
|
||
Does the magnitude of the error increase or decrease in the first step?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
radioq(["Appears to increase", "It decreases"],1,keep_order=true)
|
||
```
|
||
|
||
If $x_0$ is set near $0.75$ what happens?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
radioq(nm_choices, 2, keep_order=true)
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
Will Newton's method converge for the function $f(x) = x^5 - x + 1$ starting at $x=1$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
choices = [
|
||
"Yes",
|
||
"No. The initial guess is not close enough",
|
||
"No. The second derivative is too big",
|
||
L"No. The first derivative gets too close to $0$ for one of the $x_i$"]
|
||
answ = 2
|
||
radioq(choices, answ, keep_order=true)
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
Will Newton's method converge for the function $f(x) = 4x^5 - x + 1$ starting at $x=1$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
choices = [
|
||
"Yes",
|
||
"No. The initial guess is not close enough",
|
||
"No. The second derivative is too big, or does not exist",
|
||
L"No. The first derivative gets too close to $0$ for one of the $x_i$"]
|
||
answ = 2
|
||
radioq(choices, answ, keep_order=true)
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
Will Newton's method converge for the function $f(x) = x^{10} - 2x^3 - x + 1$ starting from $0.25$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
choices = [
|
||
"Yes",
|
||
"No. The initial guess is not close enough",
|
||
"No. The second derivative is too big, or does not exist",
|
||
L"No. The first derivative gets too close to $0$ for one of the $x_i$"]
|
||
answ = 1
|
||
radioq(choices, answ, keep_order=true)
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
Will Newton's method converge for $f(x) = 20x/(100 x^2 + 1)$ starting at $0.1$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
choices = [
|
||
"Yes",
|
||
"No. The initial guess is not close enough",
|
||
"No. The second derivative is too big, or does not exist",
|
||
L"No. The first derivative gets too close to $0$ for one of the $x_i$"]
|
||
answ = 4
|
||
radioq(choices, answ, keep_order=true)
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
Will Newton's method converge to a zero for $f(x) = \sqrt{(1 - x^2)^2}$ starting at $1.0$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
choices = [
|
||
"Yes",
|
||
"No. The initial guess is not close enough",
|
||
"No. The second derivative is too big, or does not exist",
|
||
L"No. The first derivative gets too close to $0$ for one of the $x_i$"]
|
||
answ = 3
|
||
radioq(choices, answ, keep_order=true)
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
Use Newton's method to find a root of $f(x) = 4x^4 - 5x^3 + 4x^2 -20x -6$ starting at $x_0 = 0$.
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
f(x) = 4x^4 - 5x^3 + 4x^2 -20x -6
|
||
val = find_zero((f,f') , 0, Roots.Newton())
|
||
numericq(val)
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
Use Newton's method to find a zero of $f(x) = \sin(x) - x/2$ that is *bigger* than $0$.
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
f(x) = sin(x) - x/2
|
||
val = find_zero((f,f'), 2, Roots.Newton())
|
||
numericq(val)
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
The Newton baffler (defined below) is so named, as Newton's method will fail to find the root for most starting points.
|
||
|
||
|
||
```{julia}
|
||
function newton_baffler(x)
|
||
if ( x - 0.0 ) < -0.25
|
||
0.75 * ( x - 0 ) - 0.3125
|
||
elseif ( x - 0 ) < 0.25
|
||
2.0 * ( x - 0 )
|
||
else
|
||
0.75 * ( x - 0 ) + 0.3125
|
||
end
|
||
end
|
||
```
|
||
|
||
Will Newton's method find the zero at $0.0$ starting at $1$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
yesnoq("no")
|
||
```
|
||
|
||
Consider the graph in $fig-newton-baffler-minus-1-point-1-to-1-point-1.
|
||
|
||
::: {#fig-newton-baffler-minus-1-point-1-to-1-point-1}
|
||
```{julia}
|
||
#| hold: true
|
||
plot(newton_baffler, -1.1, 1.1; label="newton baffler")
|
||
plot!(zero; label="zero")
|
||
```
|
||
|
||
Plot of Newton baffler function
|
||
:::
|
||
|
||
Starting with $x_0=1$, you can see why Newton's method will fail. Why?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
choices = [
|
||
L"It doesn't fail, it converges to $0$",
|
||
L"The tangent lines for $|x| > 0.25$ intersect at $x$ values with $|x| > 0.25$",
|
||
L"The first derivative is $0$ at $1$"
|
||
]
|
||
answ = 2
|
||
radioq(choices, answ)
|
||
```
|
||
|
||
This function does not have a small first derivative; or a large second derivative; and the bump up can be made as close to the origin as desired, so the starting point can be very close to the zero. However, even though the conditions of the error term are satisfied, the error term does not apply, as $f$ is not continuously differentiable.
|
||
|
||
|
||
###### Question
|
||
|
||
|
||
Let $f(x) = \sin(x) - x/4$. Starting at $x_0 = 2\pi$ Newton's method will converge to a value, but it will take many steps. Using a `tracks` argument to find how many steps it takes.
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
f(x) = sin(x) - x/4
|
||
x₀ = 2π
|
||
tracks = Roots.Tracks()
|
||
find_zero((f,f'), x₀, Roots.Newton(); tracks=tracks)
|
||
val = tracks.steps
|
||
numericq(val, 2)
|
||
```
|
||
|
||
What is the zero that is found?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
val = Roots.newton(f,f', 2pi)
|
||
numericq(val)
|
||
```
|
||
|
||
Is this the closest zero to the starting point, $x_0$?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
yesnoq("no")
|
||
```
|
||
|
||
###### Question
|
||
|
||
|
||
Quadratic convergence of Newton's method only applies to *simple* roots. For example, we can see (using a `tracks` argument), that it only takes $4$ steps to find a zero to $f(x) = \cos(x) - x$ starting at $x_0 = 1$. But it takes many more steps to find the same zero for $f(x) = (\cos(x) - x)^2$.
|
||
|
||
|
||
How many?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
val = 24
|
||
numericq(val, 2)
|
||
```
|
||
|
||
###### Question: Implicit equations
|
||
|
||
|
||
The equation $x^2 + x\cdot y + y^2 = 1$ is a rotated ellipse and is graphed in @fig-implicit-plot-of-rotated-ellipse.
|
||
|
||
::: {#fig-implicit-plot-of-rotated-ellipse}
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
|
||
f(x,y) = x^2 + x * y + y^2 - 1
|
||
implicit_plot(f, xlims=(-2,2), ylims=(-2,2), legend=false)
|
||
```
|
||
|
||
Plot of a rotated ellipse
|
||
:::
|
||
|
||
Can we find which point on its graph has the largest $y$ value?
|
||
|
||
|
||
This would be straightforward *if* we could write $y(x) = \dots$, for then we would simply find the critical points and investigate. But we can't so easily solve for $y$ interms of $x$. However, we can use Newton's method to do so:
|
||
|
||
|
||
```{julia}
|
||
function findy(x)
|
||
fn = y -> (x^2 + x*y + y^2) - 1
|
||
fp = y -> (x + 2y)
|
||
find_zero((fn, fp), sqrt(1 - x^2), Roots.Newton())
|
||
end
|
||
```
|
||
|
||
For a *fixed* $x$, this solves for $y$ in the equation: $F(y) = x^2 + x \cdot y + y^2 - 1 = 0$. It should be that $(x,y)$ is a solution:
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
x = .75
|
||
y = findy(x)
|
||
x^2 + x*y + y^2 ## is this 1?
|
||
```
|
||
|
||
So we have a means to find $y(x)$, but it is implicit.
|
||
|
||
|
||
Using `find_zero`, find the value $x$ which maximizes `y` by finding a zero of `y'`. Use this to find the point $(x,y)$ with largest $y$ value.
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
xstar = find_zero(findy', 0.5)
|
||
ystar = findy(xstar)
|
||
choices = ["``(-0.57735, 1.15470)``",
|
||
"``(0,0)``",
|
||
"``(0, -0.57735)``",
|
||
"``(0.57735, 0.57735)``"]
|
||
answ = 1
|
||
radioq(choices, answ)
|
||
```
|
||
|
||
(Using automatic derivatives works for values identified with `find_zero` *as long as* the initial point has its type the same as that of `x`.)
|
||
|
||
|
||
###### Question
|
||
|
||
|
||
In the last problem we used an *approximate* derivative (forward difference) in place of the derivative. This can introduce an error due to the approximation. Would Newton's method still converge if the derivative in the algorithm were replaced with an approximate derivative? In general, this can often be done *but* the convergence can be *slower* and the sensitivity to a poor initial guess even greater.
|
||
|
||
|
||
Three common approximations are given by the difference quotient for a fixed $h$: $f'(x_i) \approx (f(x_i+h)-f(x_i))/h$; the secant line approximation: $f'(x_i) \approx (f(x_i) - f(x_{i-1})) / (x_i - x_{i-1})$; and the Steffensen approximation $f'(x_i) \approx (f(x_i + f(x_i)) - f(x_i)) / f(x_i)$ (using $h=f(x_i)$).
|
||
|
||
|
||
Let's revisit the $4$-step convergence of Newton's method to the root of $f(x) = 1/x - q$ when $q=0.8$. Will these methods be as fast?
|
||
|
||
|
||
Let's define the above approximations for a given `f`:
|
||
|
||
|
||
```{julia}
|
||
q₀ = 0.8
|
||
fq(x) = 1/x - q₀
|
||
secant_approx(x0,x1) = (fq(x1) - fq(x0)) / (x1 - x0)
|
||
diffq_approx(x0, h) = secant_approx(x0, x0+h)
|
||
steff_approx(x0) = diffq_approx(x0, fq(x0))
|
||
```
|
||
|
||
Then using the difference quotient would look like:
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
Δ = 1e-6
|
||
x1 = 48/17 - 32/17 * q₀
|
||
x1 = x1 - fq(x1) / diffq_approx(x1, Δ) # |x1 - xstar| = 0.003660953777242959
|
||
x1 = x1 - fq(x1) / diffq_approx(x1, Δ) # |x1 - xstar| = 1.0719137523373945e-5; etc
|
||
```
|
||
|
||
The Steffensen method would look like:
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
x1 = 48/17 - 32/17 * q₀
|
||
x1 = x1 - fq(x1) / steff_approx(x1) # |x1 - xstar| = 0.0014382105783488086
|
||
x1 = x1 - fq(x1) / steff_approx(x1) # |x1 - xstar| = 5.944935954627084e-7; etc.
|
||
```
|
||
|
||
And the secant method like:
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
Δ = 1e-6
|
||
x1 = 48/17 - 32/17 * q₀
|
||
x0 = x1 - Δ # we need two initial values
|
||
x0, x1 = x1, x1 - fq(x1) / secant_approx(x0, x1) # |x1 - xstar| = 0.00366084553494872
|
||
x0, x1 = x1, x1 - fq(x1) / secant_approx(x0, x1) # |x1 - xstar| = 0.00019811634659716582; etc.
|
||
```
|
||
|
||
Repeat each of the above algorithms until `abs(x1 - 1.25)` is `0` (which will happen for this problem, though not in general). Record the steps.
|
||
|
||
|
||
* Does the difference quotient need *more* than $4$ steps?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
yesnoq(false)
|
||
```
|
||
|
||
* Does the secant method need *more* than $4$ steps?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
yesnoq(true)
|
||
```
|
||
|
||
* Does the Steffensen method need *more* than 4 steps?
|
||
|
||
|
||
```{julia}
|
||
#| hold: true
|
||
#| echo: false
|
||
yesnoq(false)
|
||
```
|
||
|
||
All methods work quickly with this well-behaved problem. In general the convergence rates are slightly different for each, with the Steffensen method matching Newton's method and the difference quotient method being slower in general. All can be more sensitive to the initial guess.
|
||
|
||
|
||
##### Question
|