{ "hash": "cd250ccf2ec42075c4075145af124670", "result": { "markdown": "# Newton's method\n\n\n\nThis section uses these add-on packages:\n\n``` {.julia .cell-code}\nusing CalculusWithJulia\nusing Plots\nusing SymPy\nusing Roots\n```\n\n\n\n\n---\n\n\nThe Babylonian method is an algorithm to find an approximate value for $\\sqrt{k}$. It was described by the first-century Greek mathematician Hero of [Alexandria](http://en.wikipedia.org/wiki/Babylonian_method).\n\n\nThe method starts with some initial guess, called $x_0$. It then applies a formula to produce an improved guess. This is repeated until the improved guess is accurate enough or it is clear the algorithm fails to work.\n\n\nFor the Babylonian method, the next guess, $x_{i+1}$, is derived from the current guess, $x_i$. In mathematical notation, this is the updating step:\n\n\n\n$$\nx_{i+1} = \\frac{1}{2}(x_i + \\frac{k}{x_i})\n$$\n\n\nWe use this algorithm to approximate the square root of $2$, a value known to the Babylonians.\n\n\nStart with $x$, then form $x/2 + 1/x$, from this again form $x/2 + 1/x$, repeat.\n\n\nWe represent this step using a function\n\n::: {.cell execution_count=4}\n``` {.julia .cell-code}\nbabylon(x) = x/2 + 1/x\n```\n\n::: {.cell-output .cell-output-display execution_count=5}\n```\nbabylon (generic function with 1 method)\n```\n:::\n:::\n\n\nLet's look starting with $x = 2$ as a rational number:\n\n::: {.cell hold='true' execution_count=5}\n``` {.julia .cell-code}\nx₁ = babylon(2//1)\nx₁, x₁^2.0\n```\n\n::: {.cell-output .cell-output-display execution_count=6}\n```\n(3//2, 2.25)\n```\n:::\n:::\n\n\nOur estimate improved from something which squared to $4$ down to something which squares to $2.25.$ A big improvement, but there is still more to come. Had we done one more step:\n\n::: {.cell execution_count=6}\n``` {.julia .cell-code}\nx₂ = (babylon ∘ babylon)(2//1)\nx₂, x₂^2.0\n```\n\n::: {.cell-output .cell-output-display execution_count=7}\n```\n(17//12, 2.0069444444444446)\n```\n:::\n:::\n\n\nWe now see accuracy until the third decimal point.\n\n::: {.cell execution_count=7}\n``` {.julia .cell-code}\nx₃ = (babylon ∘ babylon ∘ babylon)(2//1)\nx₃, x₃^2.0\n```\n\n::: {.cell-output .cell-output-display execution_count=8}\n```\n(577//408, 2.000006007304883)\n```\n:::\n:::\n\n\nThis is now accurate to the sixth decimal point. That is about as far as we, or the Bablyonians, would want to go by hand. Using rational numbers quickly grows out of hand. The next step shows the explosion.\n\n::: {.cell execution_count=8}\n``` {.julia .cell-code}\nreduce((x,step) -> babylon(x), 1:4, init=2//1)\n```\n\n::: {.cell-output .cell-output-display execution_count=9}\n```\n665857//470832\n```\n:::\n:::\n\n\n(In the above, we used `reduce` to repeat a function call $4$ times, as an alternative to the composition operation. In this section we show a few styles to do this repetition before introducing a packaged function.)\n\n\nHowever, with the advent of floating point numbers, the method stays quite manageable:\n\n::: {.cell hold='true' execution_count=9}\n``` {.julia .cell-code}\nxₙ = reduce((x, step) -> babylon(x), 1:6, init=2.0)\nxₙ, xₙ^2\n```\n\n::: {.cell-output .cell-output-display execution_count=10}\n```\n(1.414213562373095, 1.9999999999999996)\n```\n:::\n:::\n\n\nWe can see that the algorithm - to the precision offered by floating point numbers - has resulted in an answer `1.414213562373095`. This answer is an *approximation* to the actual answer. Approximation is necessary, as $\\sqrt{2}$ is an irrational number and so can never be exactly represented in floating point. That being said, we can see that the value of $f(x)$ is accurate to the last decimal place, so our approximation is very close and is achieved in a few steps.\n\n\n## Newton's generalization\n\n\nLet $f(x) = x^3 - 2x -5$. The value of $2$ is almost a zero, but not quite, as $f(2) = -1$. We can check that there are no *rational* roots. Though there is a method to solve the cubic it may be difficult to compute and will not be as generally applicable as some algorithm like the Babylonian method to produce an approximate answer.\n\n\nIs there some generalization to the Babylonian method?\n\n\nWe know that the tangent line is a good approximation to the function at the point. Looking at this graph gives a hint as to an algorithm:\n\n::: {.cell hold='true' execution_count=10}\n\n::: {.cell-output .cell-output-display execution_count=11}\n![](newtons_method_files/figure-html/cell-11-output-1.svg){}\n:::\n:::\n\n\nThe tangent line and the function nearly agree near $2$. So much so, that the intersection point of the tangent line with the $x$ axis nearly hides the actual zero of $f(x)$ that is near $2.1$.\n\n\nThat is, it seems that the intersection of the tangent line and the $x$ axis should be an improved approximation for the zero of the function.\n\n\nLet $x_0$ be $2$, and $x_1$ be the intersection point of the tangent line at $(x_0, f(x_0))$ with the $x$ axis. Then by the definition of the tangent line:\n\n\n\n$$\nf'(x_0) = \\frac{\\Delta y }{\\Delta x} = \\frac{f(x_0)}{x_0 - x_1}.\n$$\n\n\nThis can be solved for $x_1$ to give $x_1 = x_0 - f(x_0)/f'(x_0)$. In general, if we had $x_i$ and used the intersection point of the tangent line to produce $x_{i+1}$ we would have Newton's method:\n\n\n\n$$\nx_{i+1} = x_i - \\frac{f(x_i)}{f'(x_i)}.\n$$\n\n\nUsing automatic derivatives, as brought in with the `CalculusWithJulia` package, we can implement this algorithm.\n\n\nThe algorithm above starts at $2$ and then becomes:\n\n::: {.cell execution_count=11}\n``` {.julia .cell-code}\nf(x) = x^3 - 2x - 5\nx0 = 2.0\nx1 = x0 - f(x0) / f'(x0)\n```\n\n::: {.cell-output .cell-output-display execution_count=12}\n```\n2.1\n```\n:::\n:::\n\n\nWe can see we are closer to a zero:\n\n::: {.cell execution_count=12}\n``` {.julia .cell-code}\nf(x0), f(x1)\n```\n\n::: {.cell-output .cell-output-display execution_count=13}\n```\n(-1.0, 0.06100000000000083)\n```\n:::\n:::\n\n\nTrying again, we have\n\n::: {.cell execution_count=13}\n``` {.julia .cell-code}\nx2 = x1 - f(x1)/ f'(x1)\nx2, f(x2), f(x1)\n```\n\n::: {.cell-output .cell-output-display execution_count=14}\n```\n(2.094568121104185, 0.00018572317327247845, 0.06100000000000083)\n```\n:::\n:::\n\n\nAnd again:\n\n::: {.cell execution_count=14}\n``` {.julia .cell-code}\nx3 = x2 - f(x2)/ f'(x2)\nx3, f(x3), f(x2)\n```\n\n::: {.cell-output .cell-output-display execution_count=15}\n```\n(2.094551481698199, 1.7397612239733462e-9, 0.00018572317327247845)\n```\n:::\n:::\n\n\n::: {.cell execution_count=15}\n``` {.julia .cell-code}\nx4 = x3 - f(x3)/ f'(x3)\nx4, f(x4), f(x3)\n```\n\n::: {.cell-output .cell-output-display execution_count=16}\n```\n(2.0945514815423265, -8.881784197001252e-16, 1.7397612239733462e-9)\n```\n:::\n:::\n\n\nWe see now that $f(x_4)$ is within machine tolerance of $0$, so we call $x_4$ an *approximate zero* of $f(x)$.\n\n\n> **Newton's method:** Let $x_0$ be an initial guess for a zero of $f(x)$. Iteratively define $x_{i+1}$ in terms of the just generated $x_i$ by:\n>\n> $$\n> x_{i+1} = x_i - f(x_i) / f'(x_i).\n> $$\n>\n> Then for reasonable functions and reasonable initial guesses, the sequence of points converges to a zero of $f$.\n\n\n\nOn the computer, we know that actual convergence will likely never occur, but accuracy to a certain tolerance can often be achieved.\n\n\nIn the example above, we kept track of the previous values. This is unnecessary if only the answer is sought. In that case, the update step could use the same variable. Here we use `reduce`:\n\n::: {.cell hold='true' execution_count=16}\n``` {.julia .cell-code}\nxₙ = reduce((x, step) -> x - f(x)/f'(x), 1:4, init=2)\nxₙ, f(xₙ)\n```\n\n::: {.cell-output .cell-output-display execution_count=17}\n```\n(2.0945514815423265, -8.881784197001252e-16)\n```\n:::\n:::\n\n\nIn practice, the algorithm is implemented not by repeating the update step a fixed number of times, rather by repeating the step until either we converge or it is clear we won't converge. For good guesses and most functions, convergence happens quickly.\n\n\n:::{.callout-note}\n## Note\nNewton looked at this same example in 1699 (B.T. Polyak, *Newton's method and its use in optimization*, European Journal of Operational Research. 02/2007; 181(3):1086-1096.) though his technique was slightly different as he did not use the derivative, *per se*, but rather an approximation based on the fact that his function was a polynomial (though identical to the derivative). Raphson (1690) proposed the general form, hence the usual name of the Newton-Raphson method.\n\n:::\n\n#### Examples\n\n\n##### Example: visualizing convergence\n\n\nThis graphic demonstrates the method and the rapid convergence:\n\n\n\n::: {.cell cache='true' hold='true' execution_count=18}\n\n::: {.cell-output .cell-output-display execution_count=19}\n```{=html}\n
\"A\n

Illustration of Newton's Method converging to a zero of a function.

\n
\n
\n
\n```\n:::\n:::\n\n\n---\n\nThis interactive graphic (built using [JSXGraph](https://jsxgraph.uni-bayreuth.de/wp/index.html)) allows the adjustment of the point `x0`, initially at $0.85$. Five iterations of Newton's method are illustrated. Different positions of `x0` clearly converge, others will not.\n\n\n\n```{=html}\n
\n```\n\n```{ojs}\n//| echo: false\n//| output: false\n\nJXG = require(\"jsxgraph\");\n\n// newton's method\n\nb = JXG.JSXGraph.initBoard('jsxgraph', {\n boundingbox: [-3,5,3,-5], axis:true\n});\n\n\nf = function(x) {return x*x*x*x*x - x - 1};\nfp = function(x) { return 4*x*x*x*x - 1};\nx0 = 0.85;\n\nnm = function(x) { return x - f(x)/fp(x);};\n\nl = b.create('point', [-1.5,0], {name:'', size:0});\nr = b.create('point', [1.5,0], {name:'', size:0});\nxaxis = b.create('line', [l,r])\n\n\nP0 = b.create('glider', [x0,0,xaxis], {name:'x0'});\nP0a = b.create('point', [function() {return P0.X();},\n\t\t\t function() {return f(P0.X());}], {name:''});\n\nP1 = b.create('point', [function() {return nm(P0.X());},\n\t\t\t 0], {name:''});\nP1a = b.create('point', [function() {return P1.X();},\n\t\t\t function() {return f(P1.X());}], {name:''});\n\nP2 = b.create('point', [function() {return nm(P1.X());},\n\t\t\t 0], {name:''});\nP2a = b.create('point', [function() {return P2.X();},\n\t\t\t function() {return f(P2.X());}], {name:''});\n\nP3 = b.create('point', [function() {return nm(P2.X());},\n\t\t\t 0], {name:''});\nP3a = b.create('point', [function() {return P3.X();},\n\t\t\t function() {return f(P3.X());}], {name:''});\n\nP4 = b.create('point', [function() {return nm(P3.X());},\n\t\t\t 0], {name:''});\nP4a = b.create('point', [function() {return P4.X();},\n\t\t\t function() {return f(P4.X());}], {name:''});\nP5 = b.create('point', [function() {return nm(P4.X());},\n\t\t\t 0], {name:'x5', strokeColor:'black'});\n\n\n\n\n\nP0a.setAttribute({fixed:true});\nP1.setAttribute({fixed:true});\nP1a.setAttribute({fixed:true});\nP2.setAttribute({fixed:true});\nP2a.setAttribute({fixed:true});\nP3.setAttribute({fixed:true});\nP3a.setAttribute({fixed:true});\nP4.setAttribute({fixed:true});\nP4a.setAttribute({fixed:true});\nP5.setAttribute({fixed:true});\n\nsc = '#000000';\nb.create('segment', [P0,P0a], {strokeColor:sc, strokeWidth:1});\nb.create('segment', [P0a, P1], {strokeColor:sc, strokeWidth:1});\nb.create('segment', [P1,P1a], {strokeColor:sc, strokeWidth:1});\nb.create('segment', [P1a, P2], {strokeColor:sc, strokeWidth:1});\nb.create('segment', [P2,P2a], {strokeColor:sc, strokeWidth:1});\nb.create('segment', [P2a, P3], {strokeColor:sc, strokeWidth:1});\nb.create('segment', [P3,P3a], {strokeColor:sc, strokeWidth:1});\nb.create('segment', [P3a, P4], {strokeColor:sc, strokeWidth:1});\nb.create('segment', [P4,P4a], {strokeColor:sc, strokeWidth:1});\nb.create('segment', [P4a, P5], {strokeColor:sc, strokeWidth:1});\n\nb.create('functiongraph', [f, -1.5, 1.5])\n\n```\n\n\n##### Example: numeric not algebraic\n\n\nFor the function $f(x) = \\cos(x) - x$, we see that SymPy can not solve symbolically for a zero:\n\n::: {.cell execution_count=19}\n``` {.julia .cell-code}\n@syms x::real\nsolve(cos(x) - x, x)\n```\n\n::: {.cell-output .cell-output-error}\n```\nLoadError: PyError ($(Expr(:escape, :(ccall(#= /Users/verzani/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) \nNotImplementedError('multiple generators [x, cos(x)]\\nNo algorithms are implemented to solve equation -x + cos(x)')\n File \"/Users/verzani/.julia/conda/3/lib/python3.7/site-packages/sympy/solvers/solvers.py\", line 1106, in solve\n solution = _solve(f[0], *symbols, **flags)\n File \"/Users/verzani/.julia/conda/3/lib/python3.7/site-packages/sympy/solvers/solvers.py\", line 1720, in _solve\n raise NotImplementedError('\\n'.join([msg, not_impl_msg % f]))\n```\n:::\n:::\n\n\nWe can find a numeric solution, even though there is no closed-form answer. Here we try Newton's method:\n\n::: {.cell hold='true' execution_count=20}\n``` {.julia .cell-code}\nf(x) = cos(x) - x\nx = .5\nx = x - f(x)/f'(x) # 0.7552224171056364\nx = x - f(x)/f'(x) # 0.7391416661498792\nx = x - f(x)/f'(x) # 0.7390851339208068\nx = x - f(x)/f'(x) # 0.7390851332151607\nx = x - f(x)/f'(x)\nx, f(x)\n```\n\n::: {.cell-output .cell-output-display execution_count=21}\n```\n(0.7390851332151607, 0.0)\n```\n:::\n:::\n\n\nTo machine tolerance the answer is a zero, even though the exact answer is irrational and all finite floating point values can be represented as rational numbers.\n\n\n##### Example\n\n\nUse Newton's method to find the *largest* real solution to $e^x = x^6$.\n\n\nA plot shows us roughly where the value lies:\n\n::: {.cell hold='true' execution_count=21}\n``` {.julia .cell-code}\nf(x) = exp(x)\ng(x) = x^6\nplot(f, 0, 25, label=\"f\")\nplot!(g, label=\"g\")\n```\n\n::: {.cell-output .cell-output-display execution_count=22}\n![](newtons_method_files/figure-html/cell-22-output-1.svg){}\n:::\n:::\n\n\nClearly by $20$ the two paths diverge. We know exponentials eventually grow faster than powers, and this is seen in the graph.\n\n\nTo use Newton's method to find the intersection point. Stop when the increment $f(x)/f'(x)$ is smaller than `1e-4`. We need to turn the solution to an equation into a value where a function is $0$. Just moving the terms to one side of the equals sign gives $e^x - x^6 = 0$, or the $x$ we seek is a solution to $h(x)=0$ with $h(x) = e^x - x^6$.\n\n::: {.cell hold='true' term='true' execution_count=22}\n``` {.julia .cell-code}\nh(x) = exp(x) - x^6\nx = 20\nfor step in 1:10\n delta = h(x)/h'(x)\n x = x - delta\n @show step, x, delta\nend\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n(step, x, delta) = (1, 19.096144519894025, 0.9038554801059746)\n(step, x, delta) = (2, 18.279618508448504, 0.8165260114455216)\n(step, x, delta) = (3, 17.615584160589165, 0.6640343478593392)\n(step, x, delta) = (4, 17.186229687634423, 0.42935447295474183)\n(step, x, delta) = (5, 17.02052242498673, 0.1657072626476933)\n(step, x, delta) = (6, 16.999206949504874, 0.021315475481855334)\n(step, x, delta) = (7, 16.998887423017564, 0.00031952648730976816)\n(step, x, delta) = (8, 16.998887352296055, 7.072150775034742e-8)\n(step, x, delta) = (9, 16.99888735229605, 2.3862112429152903e-15)\n(step, x, delta) = (10, 16.99888735229605, -1.1931056214576512e-15)\n```\n:::\n:::\n\n\nSo it takes $8$ steps to get an increment that small and about `10` steps to get to full convergence.\n\n\n##### Example division as multiplication\n\n\n[Newton-Raphson Division](http://tinyurl.com/kjj9w92) is a means to divide by multiplying.\n\n\nWhy would you want to do that? Well, even for computers division is harder (read slower) than multiplying. The trick is that $p/q$ is simply $p \\cdot (1/q)$, so finding a means to compute a reciprocal by multiplying will reduce division to multiplication.\n\n\nWell suppose we have $q$, we could try to use Newton's method to find $1/q$, as it is a solution to $f(x) = x - 1/q$. The Newton update step simplifies to:\n\n\n\n$$\nx - f(x) / f'(x) \\quad\\text{or}\\quad x - (x - 1/q)/ 1 = 1/q\n$$\n\n\nThat doesn't really help, as Newton's method is just $x_{i+1} = 1/q$. That is, it just jumps to the answer, the one we want to compute by some other means!\n\n\nTrying again, we simplify the update step for a related function: $f(x) = 1/x - q$ with $f'(x) = -1/x^2$ and then one step of the process is:\n\n\n\n$$\nx_{i+1} = x_i - (1/x_i - q)/(-1/x_i^2) = -qx^2_i + 2x_i.\n$$\n\n\nNow for $q$ in the interval $[1/2, 1]$ we want to get a *good* initial guess. Here is a claim. We can use $x_0=48/17 - 32/17 \\cdot q$. Let's check graphically that this is a reasonable initial approximation to $1/q$:\n\n::: {.cell hold='true' execution_count=23}\n``` {.julia .cell-code}\nplot(q -> 1/q, 1/2, 1, label=\"1/q\")\nplot!(q -> 1/17 * (48 - 32q), label=\"linear approximation\")\n```\n\n::: {.cell-output .cell-output-display execution_count=24}\n![](newtons_method_files/figure-html/cell-24-output-1.svg){}\n:::\n:::\n\n\nIt can be shown that we have for any $q$ in $[1/2, 1]$ with initial guess $x_0 = 48/17 - 32/17\\cdot q$ that Newton's method will converge to $16$ digits in no more than this many steps:\n\n\n\n$$\n\\log_2(\\frac{53 + 1}{\\log_2(17)}).\n$$\n\n::: {.cell execution_count=24}\n``` {.julia .cell-code}\na = log2((53 + 1)/log2(17))\nceil(Integer, a)\n```\n\n::: {.cell-output .cell-output-display execution_count=25}\n```\n4\n```\n:::\n:::\n\n\nThat is $4$ steps suffices.\n\n\nFor $q = 0.80$, to find $1/q$ using the above we have\n\n::: {.cell hold='true' execution_count=25}\n``` {.julia .cell-code}\nq = 0.80\nx = (48/17) - (32/17)*q\nx = -q*x*x + 2*x\nx = -q*x*x + 2*x\nx = -q*x*x + 2*x\nx = -q*x*x + 2*x\n```\n\n::: {.cell-output .cell-output-display execution_count=26}\n```\n1.25\n```\n:::\n:::\n\n\nThis method has basically $18$ multiplication and addition operations for one division, so it naively would seem slower, but timing this shows the method is competitive with a regular division.\n\n\n## Wrapping in a function\n\n\nIn the previous examples, we saw fast convergence, guaranteed converge in $4$ steps, and an example where $8$ steps were needed to get the requested level of approximation. Newton's method usually converges quickly, but may converge slowly, and may not converge at all. Automating the task to avoid repeatedly running the update step is a task best done by the computer.\n\n\nThe `while` loop is a good way to repeat commands until some condition is met. With this, we present a simple function implementing Newton's method, we iterate until the update step gets really small (the `atol`) or the convergence takes more than $50$ steps. (There are other, better choices that could be used to determine when the algorithm should stop, these are just easy to understand.)\n\n::: {.cell execution_count=26}\n``` {.julia .cell-code}\nfunction nm(f, fp, x0)\n atol = 1e-14\n ctr = 0\n delta = Inf\n while (abs(delta) > atol) && (ctr < 50)\n delta = f(x0)/fp(x0)\n x0 = x0 - delta\n ctr = ctr + 1\n end\n\n ctr < 50 ? x0 : NaN\nend\n```\n\n::: {.cell-output .cell-output-display execution_count=27}\n```\nnm (generic function with 1 method)\n```\n:::\n:::\n\n\n##### Examples\n\n\n * Find a zero of $\\sin(x)$ starting at $x_0=3$:\n\n::: {.cell execution_count=27}\n``` {.julia .cell-code}\nnm(sin, cos, 3)\n```\n\n::: {.cell-output .cell-output-display execution_count=28}\n```\n3.141592653589793\n```\n:::\n:::\n\n\nThis is an approximation for $\\pi$, that historically found use, as the convergence is fast.\n\n\n * Find a solution to $x^5 = 5^x$ near $2$:\n\n\nWriting a function to handle this, we have:\n\n::: {.cell execution_count=28}\n``` {.julia .cell-code}\nk(x) = x^5 - 5^x\n```\n\n::: {.cell-output .cell-output-display execution_count=29}\n```\nk (generic function with 1 method)\n```\n:::\n:::\n\n\nWe could find the derivative by hand, but use the automatic one instead:\n\n::: {.cell execution_count=29}\n``` {.julia .cell-code}\nalpha = nm(k, k', 2)\nalpha, f(alpha)\n```\n\n::: {.cell-output .cell-output-display execution_count=30}\n```\n(1.764921914525776, 5.8411162339467655)\n```\n:::\n:::\n\n\n### Functions in the Roots package\n\n\nTyping in the `nm` function might be okay once, but would be tedious if it was needed each time. Besides, it isn't as robust to different inputs as possible. The `Roots` package provides a `Newton` method for `find_zero`.\n\n\nTo use a different method with `find_zero`, the calling pattern is `find_zero(f, x, M)` where `f` represent the function(s), `x` the initial point(s), and `M` the method. Here we have:\n\n::: {.cell execution_count=30}\n``` {.julia .cell-code}\nfind_zero((sin, cos), 3, Roots.Newton())\n```\n\n::: {.cell-output .cell-output-display execution_count=31}\n```\n3.141592653589793\n```\n:::\n:::\n\n\nOr, if a derivative is not specified, one can be computed using automatic differentiation:\n\n::: {.cell hold='true' execution_count=31}\n``` {.julia .cell-code}\nf(x) = sin(x)\nfind_zero((f, f'), 2, Roots.Newton())\n```\n\n::: {.cell-output .cell-output-display execution_count=32}\n```\n3.141592653589793\n```\n:::\n:::\n\n\nThe argument `verbose=true` will force a print out of a message summarizing the convergence and showing each step.\n\n::: {.cell hold='true' execution_count=32}\n``` {.julia .cell-code}\nf(x) = exp(x) - x^4\nfind_zero((f,f'), 8, Roots.Newton(); verbose=true)\n```\n\n::: {.cell-output .cell-output-display}\n```\nResults of univariate zero finding:\n\n* Converged to: 8.6131694564414\n* Algorithm: Roots.Newton()\n* iterations: 8\n* function evaluations ≈ 16\n* stopped as x_n ≈ x_{n-1} using atol=xatol, rtol=xrtol\n\nTrace:\nx₁ = 8,\t fx₁ = -1115.0420129582717\nx₂ = 9.1951685161021075,\t fx₂ = 2700.5339924159998\nx₃ = 8.7944708020788642,\t fx₃ = 615.76735783715776\nx₄ = 8.6356413896038262,\t fx₄ = 67.416233795479457\nx₅ = 8.6135576545997825,\t fx₅ = 1.1446528211590703\nx₆ = 8.6131695743314562,\t fx₆ = 0.00034750863687804667\nx₇ = 8.6131694564414101,\t fx₇ = 3.3651303965598345e-11\nx₈ = 8.6131694564413994,\t fx₈ = 1.8189894035458565e-12\nx₉ = 8.6131694564413994,\t fx₉ = 1.8189894035458565e-12\n\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=33}\n```\n8.6131694564414\n```\n:::\n:::\n\n\n##### Example: intersection of two graphs\n\n\nFind the intersection point between $f(x) = \\cos(x)$ and $g(x) = 5x$ near $0$.\n\n\nWe have Newton's method to solve for zeros of $f(x)$, i.e. when $f(x) = 0$. Here we want to solve for $x$ with $f(x) = g(x)$. To do so, we make a new function $h(x) = f(x) - g(x)$, that is $0$ when $f(x)$ equals $g(x)$:\n\n::: {.cell hold='true' execution_count=33}\n``` {.julia .cell-code}\nf(x) = cos(x)\ng(x) = 5x\nh(x) = f(x) - g(x)\nx0 = find_zero((h,h'), 0, Roots.Newton())\nx0, h(x0), f(x0), g(x0)\n```\n\n::: {.cell-output .cell-output-display execution_count=34}\n```\n(0.19616428118784215, 0.0, 0.9808214059392107, 0.9808214059392107)\n```\n:::\n:::\n\n\n---\n\n\nWe redo the above using a *parameter* for the $5$, as there are some options on how it would be done. We let `f(x,p) = cos(x) - p*x`. Then we can use `Roots.Newton` by also defining a derivative:\n\n::: {.cell hold='true' execution_count=34}\n``` {.julia .cell-code}\nf(x,p) = cos(x) - p*x\nfp(x,p) = -sin(x) - p\nxn = find_zero((f,fp), pi/4, Roots.Newton(); p=5)\nxn, f(xn, 5)\n```\n\n::: {.cell-output .cell-output-display execution_count=35}\n```\n(0.19616428118784215, 0.0)\n```\n:::\n:::\n\n\nTo use automatic differentiation is not straightforward, as we must hold the `p` fixed. For this, we introduce a closure that fixes `p` and differentiates in the `x` variable (called `u` below):\n\n::: {.cell hold='true' execution_count=35}\n``` {.julia .cell-code}\nf(x,p) = cos(x) - p*x\nfp(x,p) = (u -> f(u,p))'(x)\nxn = find_zero((f,fp), pi/4, Roots.Newton(); p=5)\n```\n\n::: {.cell-output .cell-output-display execution_count=36}\n```\n0.19616428118784215\n```\n:::\n:::\n\n\n##### Example: Finding $c$ in Rolle's Theorem\n\n\nThe function $r(x) = \\sqrt{1 - \\cos(x^2)^2}$ has a zero at $0$ and one at $a$ near $1.77$.\n\n::: {.cell execution_count=36}\n``` {.julia .cell-code}\nr(x) = sqrt(1 - cos(x^2)^2)\nplot(r, 0, 1.77)\n```\n\n::: {.cell-output .cell-output-display execution_count=37}\n![](newtons_method_files/figure-html/cell-37-output-1.svg){}\n:::\n:::\n\n\nAs $f(x)$ is differentiable between $0$ and $a$, Rolle's theorem says there will be value where the derivative is $0$. Find that value.\n\n\nThis value will be a zero of the derivative. A graph shows it should be near $1.2$, so we use that as a starting value to get the answer:\n\n::: {.cell execution_count=37}\n``` {.julia .cell-code}\nfind_zero((r',r''), 1.2, Roots.Newton())\n```\n\n::: {.cell-output .cell-output-display execution_count=38}\n```\n1.2533141373155003\n```\n:::\n:::\n\n\n## Convergence rates\n\n\nNewton's method is famously known to have \"quadratic convergence.\" What does this mean? Let the error in the $i$th step be called $e_i = x_i - \\alpha$. Then Newton's method satisfies a bound of the type:\n\n\n\n$$\n\\lvert e_{i+1} \\rvert \\leq M_i \\cdot e_i^2.\n$$\n\n\nIf $M$ were just a constant and we suppose $e_0 = 10^{-1}$ then $e_1$ would be less than $M 10^{-2}$ and $e_2$ less than $M^2 10^{-4}$, $e_3$ less than $M^3 10^{-8}$ and $e_4$ less than $M^4 10^{-16}$ which for $M=1$ is basically the machine precision when values are near $1$. That is for some problems, with a good initial guess it will take around $4$ or so steps to converge.\n\n\nTo identify $M$, let $\\alpha$ be the zero of $f$ to be approximated. Assume\n\n\n * The function $f$ has at continuous second derivative in a neighborhood of $\\alpha$.\n * The value $f'(\\alpha)$ is *non-zero* in the neighborhood of $\\alpha$.\n\n\nThen this linearization holds at each $x_i$ in the above neighborhood:\n\n\n\n$$\nf(x) = f(x_i) + f'(x_i) \\cdot (x - x_i) + \\frac{1}{2} f''(\\xi) \\cdot (x-x_i)^2.\n$$\n\n\nThe value $\\xi$ is from the mean value theorem and is between $x$ and $x_i$.\n\n\nDividing by $f'(x_i)$ and setting $x=\\alpha$ (as $f(\\alpha)=0$) leaves\n\n\n\n$$\n0 = \\frac{f(x_i)}{f'(x_i)} + (\\alpha-x_i) + \\frac{1}{2}\\cdot \\frac{f''(\\xi)}{f'(x_i)} \\cdot (\\alpha-x_i)^2.\n$$\n\n\nFor this value, we have\n\n\n\n$$\n\\begin{align*}\nx_{i+1} - \\alpha\n&= \\left(x_i - \\frac{f(x_i)}{f'(x_i)}\\right) - \\alpha\\\\\n&= \\left(x_i - \\alpha \\right) - \\frac{f(x_i)}{f'(x_i)}\\\\\n&= (x_i - \\alpha) + \\left(\n(\\alpha - x_i) + \\frac{1}{2}\\frac{f''(\\xi) \\cdot(\\alpha - x_i)^2}{f'(x_i)}\n\\right)\\\\\n&= \\frac{1}{2}\\frac{f''(\\xi)}{f'(x_i)} \\cdot(x_i - \\alpha)^2.\n\\end{align*}\n$$\n\n\nThat is\n\n\n\n$$\ne_{i+1} = \\frac{1}{2}\\frac{f''(\\xi)}{f'(x_i)} e_i^2.\n$$\n\n\nThis convergence to $\\alpha$ will be quadratic *if*:\n\n\n * The initial guess $x_0$ is not too far from $\\alpha$, so $e_0$ is managed.\n * The derivative at $\\alpha$ is not too close to $0$, hence, by continuity $f'(x_i)$ is not too close to $0$. (As it appears in the denominator). That is, the function can't be too flat, which should make sense, as then the tangent line is nearly parallel to the $x$ axis and would intersect far away.\n * The function $f$ has a continuous second derivative at $\\alpha$.\n * The second derivative is not too big (in absolute value) near $\\alpha$. A large second derivative means the function is very concave, which means it is \"turning\" a lot. In this case, the function turns away from the tangent line quickly, so the tangent line's zero is not necessarily a good approximation to the actual zero, $\\alpha$.\n\n\n:::{.callout-note}\n## Note\nThe basic tradeoff: methods like Newton's are faster than the bisection method in terms of function calls, but are not guaranteed to converge, as the bisection method is.\n\n:::\n\nWhat can go wrong when one of these isn't the case is illustrated next:\n\n\n### Poor initial step\n\n::: {.cell cache='true' hold='true' execution_count=38}\n\n::: {.cell-output .cell-output-display execution_count=39}\n```{=html}\n
\"A\n

Illustration of Newton's Method converging to a zero of a function, but slowly as the initial guess, is very poor, and not close to the zero. The algorithm does converge in this illustration, but not quickly and not to the nearest root from the initial guess.

\n
\n
\n
\n```\n:::\n:::\n\n\n::: {.cell cache='true' hold='true' execution_count=39}\n\n::: {.cell-output .cell-output-display execution_count=40}\n```{=html}\n
\"A\n

Illustration of Newton's method failing to coverge as for some \\(x_i\\), \\(f'(x_i)\\) is too close to \\(0\\). In this instance after a few steps, the algorithm just cycles around the local minimum near \\(0.66\\). The values of \\(x_i\\) repeat in the pattern: \\(1.0002, 0.7503, -0.0833, 1.0002, \\dots\\). This is also an illustration of a poor initial guess. If there is a local minimum or maximum between the guess and the zero, such cycles can occur.

\n
\n
\n
\n```\n:::\n:::\n\n\n### The second derivative is too big\n\n::: {.cell cache='true' hold='true' execution_count=40}\n\n::: {.cell-output .cell-output-display execution_count=41}\n```{=html}\n
\"A\n

Illustration of Newton's Method not converging. Here the second derivative is too big near the zero - it blows up near \\(0\\) - and the convergence does not occur. Rather the iterates increase in their distance from the zero.

\n
\n
\n
\n```\n:::\n:::\n\n\n### The tangent line at some xᵢ is flat\n\n::: {.cell cache='true' hold='true' execution_count=41}\n\n::: {.cell-output .cell-output-display execution_count=42}\n```{=html}\n
\"A\n

The function \\(f(x) = x^{20} - 1\\) has two bad behaviours for Newton's method: for \\(x < 1\\) the derivative is nearly \\(0\\) and for \\(x>1\\) the second derivative is very big. In this illustration, we have an initial guess of \\(x_0=8/9\\). As the tangent line is fairly flat, the next approximation is far away, \\(x_1 = 1.313\\dots\\). As this guess is is much bigger than \\(1\\), the ratio \\(f(x)/f'(x) \\approx x^{20}/(20x^{19}) = x/20\\), so \\(x_i - x_{i-1} \\approx (19/20)x_i\\) yielding slow, linear convergence until \\(f''(x_i)\\) is moderate. For this function, starting at \\(x_0=8/9\\) takes 11 steps, at \\(x_0=7/8\\) takes 13 steps, at \\(x_0=3/4\\) takes \\(55\\) steps, and at \\(x_0=1/2\\) it takes \\(204\\) steps.

\n
\n
\n
\n```\n:::\n:::\n\n\n###### Example\n\n\nSuppose $\\alpha$ is a simple zero for $f(x)$. (The value $\\alpha$ is a zero of multiplicity $k$ if $f(x) = (x-\\alpha)^kg(x)$ where $g(\\alpha)$ is not zero. A simple zero has multiplicity $1$. If $f'(\\alpha) \\neq 0$ and the second derivative exists, then a zero $\\alpha$ will be simple.) Around $\\alpha$, quadratic convergence should apply. However, consider the function $g(x) = f(x)^k$ for some integer $k \\geq 2$. Then $\\alpha$ is still a zero, but the derivative of $g$ at $\\alpha$ is zero, so the tangent line is basically flat. This will slow the convergence up. We can see that the update step $g'(x)/g(x)$ becomes $(1/k) f'(x)/f(x)$, so an extra factor is introduced.\n\n\nThe calculation that produces the quadratic convergence now becomes:\n\n\n\n$$\nx_{i+1} - \\alpha = (x_i - \\alpha) - \\frac{1}{k}(x_i-\\alpha + \\frac{f''(\\xi)}{2f'(x_i)}(x_i-\\alpha)^2) =\n\\frac{k-1}{k} (x_i-\\alpha) + \\frac{f''(\\xi)}{2kf'(x_i)}(x_i-\\alpha)^2.\n$$\n\n\nAs $k > 1$, the $(x_i - \\alpha)$ term dominates, and we see the convergence is linear with $\\lvert e_{i+1}\\rvert \\approx (k-1)/k \\lvert e_i\\rvert$.\n\n\n## Questions\n\n\n###### Question\n\n\nLook at this graph with $x_0$ marked with a point:\n\n::: {.cell hold='true' execution_count=42}\n\n::: {.cell-output .cell-output-display execution_count=43}\n![](newtons_method_files/figure-html/cell-43-output-1.svg){}\n:::\n:::\n\n\nIf one step of Newton's method was used, what would be the value of $x_1$?\n\n::: {.cell hold='true' execution_count=43}\n\n::: {.cell-output .cell-output-display execution_count=44}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nLook at this graph of some increasing, concave up $f(x)$ with initial point $x_0$ marked. Let $\\alpha$ be the zero.\n\n::: {.cell hold='true' execution_count=44}\n\n::: {.cell-output .cell-output-display execution_count=45}\n![](newtons_method_files/figure-html/cell-45-output-1.svg){}\n:::\n:::\n\n\nWhat can be said about $x_1$?\n\n::: {.cell hold='true' execution_count=45}\n\n::: {.cell-output .cell-output-display execution_count=46}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n---\n\nLook at this graph of some increasing, concave up $f(x)$ with initial point $x_0$ marked. Let $\\alpha$ be the zero.\n\n::: {.cell hold='true' execution_count=46}\n\n::: {.cell-output .cell-output-display execution_count=47}\n![](newtons_method_files/figure-html/cell-47-output-1.svg){}\n:::\n:::\n\n\nWhat can be said about $x_1$?\n\n::: {.cell hold='true' execution_count=47}\n\n::: {.cell-output .cell-output-display execution_count=48}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n---\n\n\nSuppose $f(x)$ is increasing and concave up. From the tangent line representation: $f(x) = f(c) + f'(c)\\cdot(x-c) + f''(\\xi)/2 \\cdot(x-c)^2$, explain why it must be that the graph of $f(x)$ lies on or *above* the tangent line.\n\n::: {.cell hold='true' execution_count=48}\n\n::: {.cell-output .cell-output-display execution_count=49}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\nThis question can be used to give a proof for the previous two questions, which can be answered by considering the graphs alone. Combined, they say that if a function is increasing and concave up and $\\alpha$ is a zero, then if $x_0 < \\alpha$ it will be $x_1 > \\alpha$, and for any $x_i > \\alpha$, $\\alpha <= x_{i+1} <= x_\\alpha$, so the sequence in Newton's method is decreasing and bounded below; conditions for which it is guaranteed mathematically there will be convergence.\n\n\n###### Question\n\n\nLet $f(x) = x^2 - 3^x$. This has derivative $2x - 3^x \\cdot \\log(3)$. Starting with $x_0=0$, what does Newton's method converge on?\n\n::: {.cell hold='true' execution_count=49}\n\n::: {.cell-output .cell-output-display execution_count=50}\n```{=html}\n
\n
\n
\n
\n
\n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nLet $f(x) = \\exp(x) - x^4$. There are 3 zeros for this function. Which one does Newton's method converge to when $x_0=2$?\n\n::: {.cell hold='true' execution_count=50}\n\n::: {.cell-output .cell-output-display execution_count=51}\n```{=html}\n
\n
\n
\n
\n
\n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nLet $f(x) = \\exp(x) - x^4$. As mentioned, there are 3 zeros for this function. Which one does Newton's method converge to when $x_0=8$?\n\n::: {.cell hold='true' execution_count=51}\n\n::: {.cell-output .cell-output-display execution_count=52}\n```{=html}\n
\n
\n
\n
\n
\n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nLet $f(x) = \\sin(x) - \\cos(4\\cdot x)$.\n\n\nStarting at $\\pi/8$, solve for the root returned by Newton's method\n\n::: {.cell hold='true' execution_count=52}\n\n::: {.cell-output .cell-output-display execution_count=53}\n```{=html}\n
\n
\n
\n
\n
\n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nUsing Newton's method find a root to $f(x) = \\cos(x) - x^3$ starting at $x_0 = 1/2$.\n\n::: {.cell hold='true' execution_count=53}\n\n::: {.cell-output .cell-output-display execution_count=54}\n```{=html}\n
\n
\n
\n
\n
\n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nUse Newton's method to find a root of $f(x) = x^5 + x -1$. Make a quick graph to find a reasonable starting point.\n\n::: {.cell hold='true' execution_count=54}\n\n::: {.cell-output .cell-output-display execution_count=55}\n```{=html}\n
\n
\n
\n
\n
\n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\n\nFor the following graph, graphically consider the algorithm for a few different starting points.\n\n::: {.cell hold='true' execution_count=56}\n\n::: {.cell-output .cell-output-display execution_count=57}\n![](newtons_method_files/figure-html/cell-57-output-1.svg){}\n:::\n:::\n\n\nIf $x_0$ is $1$ what occurs?\n\n::: {.cell execution_count=57}\n\n::: {.cell-output .cell-output-display execution_count=58}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\nWhen $x_0 = 1.0$ the following values are true for $f$:\n\n::: {.cell execution_count=58}\n\n::: {.cell-output .cell-output-display execution_count=59}\n```\n(e₀ = -0.16730397826141874, f₀′ = 6.0, f̄₀′′ = 31.81133480963258, ē₁ = 0.0742015850567364)\n```\n:::\n:::\n\n\nWhere the values `f̄₀′′` and `ē₁` are worst-case estimates when $\\xi$ is between $x_0$ and the zero.\n\n\nDoes the magnitude of the error increase or decrease in the first step?\n\n::: {.cell hold='true' execution_count=59}\n\n::: {.cell-output .cell-output-display execution_count=60}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\nIf $x_0$ is set near $0.40$ what happens?\n\n::: {.cell hold='true' execution_count=60}\n\n::: {.cell-output .cell-output-display execution_count=61}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\nWhen $x_0 = 0.4$ the following values are true for $f$:\n\n::: {.cell hold='true' execution_count=61}\n\n::: {.cell-output .cell-output-display execution_count=62}\n```\n(e₀ = -0.7673039782614187, f₀′ = 1.1280000000000001, f̄₀′′ = 31.81133480963258, ē₁ = 8.301903808997139)\n```\n:::\n:::\n\n\nWhere the values `f̄₀′′` and `ē₁` are worst-case estimates when $\\xi$ is between $x_0$ and the zero.\n\n\nDoes the magnitude of the error increase or decrease in the first step?\n\n::: {.cell hold='true' execution_count=62}\n\n::: {.cell-output .cell-output-display execution_count=63}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\nIf $x_0$ is set near $0.75$ what happens?\n\n::: {.cell hold='true' execution_count=63}\n\n::: {.cell-output .cell-output-display execution_count=64}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nWill Newton's method converge for the function $f(x) = x^5 - x + 1$ starting at $x=1$?\n\n::: {.cell hold='true' execution_count=64}\n\n::: {.cell-output .cell-output-display execution_count=65}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nWill Newton's method converge for the function $f(x) = 4x^5 - x + 1$ starting at $x=1$?\n\n::: {.cell hold='true' execution_count=65}\n\n::: {.cell-output .cell-output-display execution_count=66}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nWill Newton's method converge for the function $f(x) = x^{10} - 2x^3 - x + 1$ starting from $0.25$?\n\n::: {.cell hold='true' execution_count=66}\n\n::: {.cell-output .cell-output-display execution_count=67}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nWill Newton's method converge for $f(x) = 20x/(100 x^2 + 1)$ starting at $0.1$?\n\n::: {.cell hold='true' execution_count=67}\n\n::: {.cell-output .cell-output-display execution_count=68}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nWill Newton's method converge to a zero for $f(x) = \\sqrt{(1 - x^2)^2}$?\n\n::: {.cell hold='true' execution_count=68}\n\n::: {.cell-output .cell-output-display execution_count=69}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nUse Newton's method to find a root of $f(x) = 4x^4 - 5x^3 + 4x^2 -20x -6$ starting at $x_0 = 0$.\n\n::: {.cell hold='true' execution_count=69}\n\n::: {.cell-output .cell-output-display execution_count=70}\n```{=html}\n
\n
\n
\n
\n
\n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nUse Newton's method to find a zero of $f(x) = \\sin(x) - x/2$ that is *bigger* than $0$.\n\n::: {.cell hold='true' execution_count=70}\n\n::: {.cell-output .cell-output-display execution_count=71}\n```{=html}\n
\n
\n
\n
\n
\n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nThe Newton baffler (defined below) is so named, as Newton's method will fail to find the root for most starting points.\n\n::: {.cell execution_count=71}\n``` {.julia .cell-code}\nfunction newton_baffler(x)\n if ( x - 0.0 ) < -0.25\n 0.75 * ( x - 0 ) - 0.3125\n elseif ( x - 0 ) < 0.25\n 2.0 * ( x - 0 )\n else\n 0.75 * ( x - 0 ) + 0.3125\n end\nend\n```\n\n::: {.cell-output .cell-output-display execution_count=72}\n```\nnewton_baffler (generic function with 1 method)\n```\n:::\n:::\n\n\nWill Newton's method find the zero at $0.0$ starting at $1$?\n\n::: {.cell hold='true' execution_count=72}\n\n::: {.cell-output .cell-output-display execution_count=73}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\nConsidering this plot:\n\n::: {.cell hold='true' execution_count=73}\n``` {.julia .cell-code}\nplot(newton_baffler, -1.1, 1.1)\n```\n\n::: {.cell-output .cell-output-display execution_count=74}\n![](newtons_method_files/figure-html/cell-74-output-1.svg){}\n:::\n:::\n\n\nStarting with $x_0=1$, you can see why Newton's method will fail. Why?\n\n::: {.cell hold='true' execution_count=74}\n\n::: {.cell-output .cell-output-display execution_count=75}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\nThis function does not have a small first derivative; or a large second derivative; and the bump up can be made as close to the origin as desired, so the starting point can be very close to the zero. However, even though the conditions of the error term are satisfied, the error term does not apply, as $f$ is not continuously differentiable.\n\n\n###### Question\n\n\nLet $f(x) = \\sin(x) - x/4$. Starting at $x_0 = 2\\pi$ Newton's method will converge to a value, but it will take many steps. Using the argument `verbose=true` for `find_zero`, how many steps does it take:\n\n::: {.cell hold='true' execution_count=75}\n\n::: {.cell-output .cell-output-display execution_count=76}\n```{=html}\n
\n
\n
\n
\n
\n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\nWhat is the zero that is found?\n\n::: {.cell hold='true' execution_count=76}\n\n::: {.cell-output .cell-output-display execution_count=77}\n```{=html}\n
\n
\n
\n
\n
\n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\nIs this the closest zero to the starting point, $x_0$?\n\n::: {.cell hold='true' execution_count=77}\n\n::: {.cell-output .cell-output-display execution_count=78}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question\n\n\nQuadratic convergence of Newton's method only applies to *simple* roots. For example, we can see (using the `verbose=true` argument to the `Roots` package's `newton` method, that it only takes $4$ steps to find a zero to $f(x) = \\cos(x) - x$ starting at $x_0 = 1$. But it takes many more steps to find the same zero for $f(x) = (\\cos(x) - x)^2$.\n\n\nHow many?\n\n::: {.cell hold='true' execution_count=78}\n\n::: {.cell-output .cell-output-display execution_count=79}\n```{=html}\n
\n
\n
\n
\n
\n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n###### Question: Implicit equations\n\n\nThe equation $x^2 + x\\cdot y + y^2 = 1$ is a rotated ellipse.\n\n::: {.cell hold='true' execution_count=79}\n\n::: {.cell-output .cell-output-display execution_count=80}\n![](newtons_method_files/figure-html/cell-80-output-1.svg){}\n:::\n:::\n\n\nCan we find which point on its graph has the largest $y$ value?\n\n\nThis would be straightforward *if* we could write $y(x) = \\dots$, for then we would simply find the critical points and investiate. But we can't so easily solve for $y$ interms of $x$. However, we can use Newton's method to do so:\n\n::: {.cell execution_count=80}\n``` {.julia .cell-code}\nfunction findy(x)\n fn = y -> (x^2 + x*y + y^2) - 1\n fp = y -> (x + 2y)\n find_zero((fn, fp), sqrt(1 - x^2), Roots.Newton())\nend\n```\n\n::: {.cell-output .cell-output-display execution_count=81}\n```\nfindy (generic function with 1 method)\n```\n:::\n:::\n\n\nFor a *fixed* x, this solves for $y$ in the equation: $F(y) = x^2 + x \\cdot y + y^2 - 1 = 0$. It should be that $(x,y)$ is a solution:\n\n::: {.cell hold='true' execution_count=81}\n``` {.julia .cell-code}\nx = .75\ny = findy(x)\nx^2 + x*y + y^2 ## is this 1?\n```\n\n::: {.cell-output .cell-output-display execution_count=82}\n```\n1.0000000000000002\n```\n:::\n:::\n\n\nSo we have a means to find $y(x)$, but it is implicit.\n\n\nUsing `find_zero`, find the value $x$ which maximizes `y` by finding a zero of `y'`. Use this to find the point $(x,y)$ with largest $y$ value.\n\n::: {.cell hold='true' execution_count=82}\n\n::: {.cell-output .cell-output-display execution_count=83}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n(Using automatic derivatives works for values identified with `find_zero` *as long as* the initial point has its type the same as that of `x`.)\n\n\n###### Question\n\n\nIn the last problem we used an *approximate* derivative (forward difference) in place of the derivative. This can introduce an error due to the approximation. Would Newton's method still converge if the derivative in the algorithm were replaced with an approximate derivative? In general, this can often be done *but* the convergence can be *slower* and the sensitivity to a poor initial guess even greater.\n\n\nThree common approximations are given by the difference quotient for a fixed $h$: $f'(x_i) \\approx (f(x_i+h)-f(x_i))/h$; the secant line approximation: $f'(x_i) \\approx (f(x_i) - f(x_{i-1})) / (x_i - x_{i-1})$; and the Steffensen approximation $f'(x_i) \\approx (f(x_i + f(x_i)) - f(x_i)) / f(x_i)$ (using $h=f(x_i)$).\n\n\nLet's revisit the $4$-step convergence of Newton's method to the root of $f(x) = 1/x - q$ when $q=0.8$. Will these methods be as fast?\n\n\nLet's define the above approximations for a given `f`:\n\n::: {.cell execution_count=83}\n``` {.julia .cell-code}\nq₀ = 0.8\nfq(x) = 1/x - q₀\nsecant_approx(x0,x1) = (fq(x1) - f(x0)) / (x1 - x0)\ndiffq_approx(x0, h) = secant_approx(x0, x0+h)\nsteff_approx(x0) = diffq_approx(x0, fq(x0))\n```\n\n::: {.cell-output .cell-output-display execution_count=84}\n```\nsteff_approx (generic function with 1 method)\n```\n:::\n:::\n\n\nThen using the difference quotient would look like:\n\n::: {.cell hold='true' execution_count=84}\n``` {.julia .cell-code}\nΔ = 1e-6\nx1 = 42/17 - 32/17 * q₀\nx1 = x1 - fq(x1) / diffq_approx(x1, Δ) # |x1 - xstar| = 0.06511395862036995\nx1 = x1 - fq(x1) / diffq_approx(x1, Δ) # |x1 - xstar| = 0.003391809999860218; etc\n```\n\n::: {.cell-output .cell-output-display execution_count=85}\n```\n0.9647072573629651\n```\n:::\n:::\n\n\nThe Steffensen method would look like:\n\n::: {.cell hold='true' execution_count=85}\n``` {.julia .cell-code}\nx1 = 42/17 - 32/17 * q₀\nx1 = x1 - fq(x1) / steff_approx(x1) # |x1 - xstar| = 0.011117056291670258\nx1 = x1 - fq(x1) / steff_approx(x1) # |x1 - xstar| = 3.502579696146313e-5; etc.\n```\n\n::: {.cell-output .cell-output-display execution_count=86}\n```\n1.0994223446163458\n```\n:::\n:::\n\n\nAnd the secant method like:\n\n::: {.cell hold='true' execution_count=86}\n``` {.julia .cell-code}\nΔ = 1e-6\nx1 = 42/17 - 32/17 * q₀\nx0 = x1 - Δ # we need two initial values\nx0, x1 = x1, x1 - fq(x1) / secant_approx(x0, x1) # |x1 - xstar| = 8.222358365284066e-6\nx0, x1 = x1, x1 - fq(x1) / secant_approx(x0, x1) # |x1 - xstar| = 1.8766323799379592e-6; etc.\n```\n\n::: {.cell-output .cell-output-display execution_count=87}\n```\n(0.9647065698627694, 0.9647070425296034)\n```\n:::\n:::\n\n\nRepeat each of the above algorithms until `abs(x1 - 1.25)` is `0` (which will happen for this problem, though not in general). Record the steps.\n\n\n * Does the difference quotient need *more* than $4$ steps?\n\n::: {.cell hold='true' execution_count=87}\n\n::: {.cell-output .cell-output-display execution_count=88}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n * Does the secant method need *more* than $4$ steps?\n\n::: {.cell hold='true' execution_count=88}\n\n::: {.cell-output .cell-output-display execution_count=89}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\n * Does the Steffensen method need *more* than 4 steps?\n\n::: {.cell hold='true' execution_count=89}\n\n::: {.cell-output .cell-output-display execution_count=90}\n```{=html}\n
\n
\n
\n
\n
\n
\n \n
\n
\n \n
\n\n \n
\n
\n
\n
\n
\n
\n\n\n```\n:::\n:::\n\n\nAll methods work quickly with this well-behaved problem. In general the convergence rates are slightly different for each, with the Steffensen method matching Newton's method and the difference quotient method being slower in general. All can be more sensitive to the initial guess.\n\n", "supporting": [ "newtons_method_files" ], "filters": [], "includes": { "include-in-header": [ "\n\n\n" ] } } }