daviddoji/CalculusWithJuliaNotes.jl
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

initial

Browse Source
This commit is contained in:
jverzani
2022-05-24 13:51:49 -04:00
parent 5c0fd1b6fe
commit 244f492f9e
240 changed files with 65211 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
CwJ/derivatives/Project.toml Normal file
View File

@@ -0,0 +1,15 @@
[deps]
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210"
IntervalArithmetic = "d1acc4aa-44c8-5952-acd4-ba5d80a2a253"
IntervalConstraintProgramming = "138f1668-1576-5ad7-91b9-7425abbf3153"
LaTeXStrings = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f"
MDBM = "dd61e66b-39ce-57b0-8813-509f78be4b4d"
Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
Printf = "de0858da-6303-5e67-8744-51eddeeeb8d7"
QuadGK = "1fd47b50-473d-5c70-9696-f719f8f3bcdc"
Roots = "f2b01f46-fcfa-551c-844a-d8ac1e96c665"
SymPy = "24249f21-da20-56a4-8eb1-6a02cf4ae2e6"
TaylorSeries = "6aa5eb33-94cf-58f4-a9d0-e4b2c4fc25ea"
TermInterface = "8ea1fca8-c5ef-4a55-8b96-4e9afe9c9a3c"
Unitful = "1986cc42-f94f-5a68-af5c-568840ba703d"

543
CwJ/derivatives/curve_sketching.jmd Normal file
View File

@@ -0,0 +1,543 @@
# Curve Sketching
This section uses the following add-on packages:
```julia
using CalculusWithJulia
using Plots
using SymPy
using Roots
using Polynomials # some name clash with SymPy
```
```julia; echo=false; results="hidden"
using CalculusWithJulia.WeaveSupport
fig_size=(600, 400)
const frontmatter = (
title = "Curve Sketching",
description = "Calculus with Julia: Curve Sketching",
tags = ["CalculusWithJulia", "derivatives", "curve sketching"],
);
nothing
```
----
The figure illustrates a means to *sketch* a sine curve - identify as
many of the following values as you can:
* asymptotic behaviour (as ``x \rightarrow \pm \infty``),
* periodic behaviour,
* vertical asymptotes,
* the $y$ intercept,
* any $x$ intercept(s),
* local peaks and valleys (relative extrema).
* concavity
With these, a sketch fills in between the
points/lines associated with these values.
```julia; hold=true; echo=false; cache=true
### {{{ sketch_sin_plot }}}
function sketch_sin_plot_graph(i)
f(x) = 10*sin(pi/2*x) # [0,4]
deltax = 1/10
deltay = 5/10
zs = find_zeros(f, 0-deltax, 4+deltax)
cps = find_zeros(D(f), 0-deltax, 4+deltax)
xs = range(0, stop=4*(i-2)/6, length=50)
if i == 1
## plot zeros
title = "Plot the zeros"
p = scatter(zs, 0*zs, title=title, xlim=(-deltax,4+deltax), ylim=(-10-deltay,10+deltay), legend=false)
elseif i == 2
## plot extrema
title = "Plot the local extrema"
p = scatter(zs, 0*zs, title=title, xlim=(-deltax,4+deltax), ylim=(-10-deltay,10+deltay), legend=false)
scatter!(p, cps, f.(cps))
else
## sketch graph
title = "sketch the graph"
p = scatter(zs, 0*zs, title=title, xlim=(-deltax,4+deltax), ylim=(-10-deltay,10+deltay), legend=false)
scatter!(p, cps, f.(cps))
plot!(p, xs, f.(xs))
end
p
end
caption = L"""
After identifying asymptotic behaviours,
a curve sketch involves identifying the $y$ intercept, if applicable; the $x$ intercepts, if possible; the local extrema; and changes in concavity. From there a sketch fills in between the points. In this example, the periodic function $f(x) = 10\cdot\sin(\pi/2\cdot x)$ is sketched over $[0,4]$.
"""
n = 8
anim = @animate for i=1:n
sketch_sin_plot_graph(i)
end
imgfile = tempname() * ".gif"
gif(anim, imgfile, fps = 1)
ImageFile(imgfile, caption)
```
Though this approach is most useful for hand-sketches, the underlying
concepts are important for properly framing graphs made with the
computer.
We can easily make a graph of a function over a specified
interval. What is not always so easy is to pick an interval that shows
off the features of interest. In the section on
[rational](../precalc/rational_functions.html) functions there was a
discussion about how to draw graphs for rational functions so that
horizontal and vertical asymptotes can be seen. These are properties
of the "large." In this section, we build on this, but concentrate now
on more local properties of a function.
##### Example
Produce a graph of the function $f(x) = x^4 -13x^3 + 56x^2-92x + 48$.
We identify this as a fourth-degree polynomial with postive leading
coefficient. Hence it will eventually look $U$-shaped. If we graph
over a too-wide interval, that is all we will see. Rather, we do some
work to produce a graph that shows the zeros, peaks, and valleys of
$f(x)$. To do so, we need to know the extent of the zeros. We can try
some theory, but instead we just guess and if that fails, will work harder:
```julia;
f(x) = x^4 - 13x^3 + 56x^2 -92x + 48
rts = find_zeros(f, -10, 10)
```
As we found $4$ roots, we know by the fundamental theorem of algebra we have them all. This means, our graph need not focus on values much larger than $6$ or much smaller than $1$.
To know where the peaks and valleys are, we look for the critical points:
```julia;
cps = find_zeros(f', 1, 6)
```
Because we have the $4$ distinct zeros, we must have the peaks and
valleys appear in an interleaving manner, so a search over $[1,6]$
finds all three critical points and without checking, they must
correspond to relative extrema.
Next we identify the *inflection points* which are among the zeros of the second derivative (when defined):
```julia
ips = find_zeros(f'', 1, 6)
```
If there is no sign change for either ``f'`` or ``f''`` over ``[a,b]`` then the sketch of ``f`` on this interval must be one of:
* increasing and concave up (if ``f' > 0`` and ``f'' > 0``)
* increasing and concave down (if ``f' > 0`` and ``f'' < 0``)
* decreasing and concave up (if ``f' < 0`` and ``f'' > 0``)
* decreasing and concave down (if ``f' < 0`` and ``f'' < 0``)
This aids in sketching the graph between the critical points and inflection points.
We finally check that if we were to just use $[0,7]$ as a domain to
plot over that the function doesn't get too large to mask the
oscillations. This could happen if the $y$ values at the end points
are too much larger than the $y$ values at the peaks and valleys, as
only so many pixels can be used within a graph. For this we have:
```julia;
f.([0, cps..., 7])
```
The values at $0$ and at $7$ are a bit large, as compared to the
relative extrema, and since we know the graph is eventually
$U$-shaped, this offers no insight. So we narrow the range a bit for
the graph:
```julia;
plot(f, 0.5, 6.5)
```
----
This sort of analysis can be automated. The plot "recipe" for polynomials from the `Polynomials` package does similar considerations to choose a viewing window:
```julia
xₚ = variable(Polynomial)
plot(f(xₚ)) # f(xₚ) of Polynomial type
```
##### Example
Graph the function
```math
f(x) = \frac{(x-1)\cdot(x-3)^2}{x \cdot (x-2)}.
```
Not much to do here if you are satisfied with a graph that only gives insight into the asymptotes of this rational function:
```julia;
𝒇(x) = ( (x-1)*(x-3)^2 ) / (x * (x-2) )
plot(𝒇, -50, 50)
```
We can see the slant asymptote and hints of vertical asymptotes, but,
we'd like to see more of the basic features of the graph.
Previously, we have discussed rational functions and their
asymptotes. This function has numerator of degree ``3`` and denominator of
degree ``2``, so will have a slant asymptote. As well, the zeros of the
denominator, $0$ and $-2$, will lead to vertical asymptotes.
To identify how wide a viewing window should be, for the rational
function the asymptotic behaviour is determined after the concavity is
done changing and we are past all relative extrema, so we should take
an interval that includes all potential inflection points and critical
points:
```julia;
𝒇cps = find_zeros(𝒇', -10, 10)
poss_ips = find_zero(𝒇'', (-10, 10))
extrema(union(𝒇cps, poss_ips))
```
So a range over $[-5,5]$ should display the key features including the slant asymptote.
Previously we used the `rangeclamp` function defined in `CalculusWithJulia` to avoid the distortion that vertical asymptotes can have:
```julia;
plot(rangeclamp(𝒇), -5, 5)
```
With this graphic, we can now clearly see in the graph the two zeros at $x=1$ and $x=3$, the vertical asymptotes at $x=0$ and $x=2$, and the slant asymptote.
----
Again, this sort of analysis can be systematized. The rational function type in the `Polynomials` package takes a stab at that, but isn't quite so good at capturing the slant asymptote:
```julia
xᵣ = variable(RationalFunction)
plot(𝒇(xᵣ)) # f(x) of RationalFunction type
```
##### Example
Consider the function ``V(t) = 170 \sin(2\pi\cdot 60 \cdot t)``, a model for the alternating current waveform for an outlet in the United States. Create a graph.
Blindly trying to graph this, we will see immediate issues:
```julia
V(t) = 170 * sin(2*pi*60*t)
plot(V, -2pi, 2pi)
```
Ahh, this periodic function is *too* rapidly oscillating to be plotted without care. We recognize this as being of the form ``V(t) = a\cdot\sin(c\cdot t)``, so where the sine function has a period of ``2\pi``, this will have a period of ``2\pi/c``, or ``1/60``. So instead of using ``(-2\pi, 2\pi)`` as the interval to plot over, we need something much smaller:
```julia
plot(V, -1/60, 1/60)
```
##### Example
Plot the function ``f(x) = \ln(x/100)/x``.
We guess that this function has a *vertical* asymptote at ``x=0+`` and a horizontal asymptote as ``x \rightarrow \infty``, we verify through:
```julia
@syms x
ex = log(x/100)/x
limit(ex, x=>0, dir="+"), limit(ex, x=>oo)
```
The ``\ln(x/100)`` part of ``f`` goes ``-\infty`` as ``x \rightarrow 0+``; yet ``f(x)`` is eventually positive as ``x \rightarrow 0``. So a graph should
* not show too much of the vertical asymptote
* capture the point where ``f(x)`` must cross ``0``
* capture the point where ``f(x)`` has a relative maximum
* show enough past this maximum to indicate to the reader the eventual horizontal asyptote.
For that, we need to get the ``x`` intercepts and the critical points. The ``x/100`` means this graph has some scaling to it, so we first look between ``0`` and ``200``:
```julia
find_zeros(ex, 0, 200) # domain is (0, oo)
```
Trying the same for the critical points comes up empty. We know there is one, but it is past ``200``. Scanning wider, we see:
```julia
find_zeros(diff(ex,x), 0, 500)
```
So maybe graphing over ``[50, 300]`` will be a good start:
```julia
plot(ex, 50, 300)
```
But it isn't! The function takes its time getting back towards ``0``. We know that there must be a change of concavity as ``x \rightarrow \infty``, as there is a horizontal asymptote. We looks for the anticipated inflection point to ensure our graph includes that:
```julia
find_zeros(diff(ex, x, x), 1, 5000)
```
So a better plot is found by going well beyond that inflection point:
```julia
plot(ex, 75, 1500)
```
## Questions
###### Question
Consider this graph
```julia; hold=true; echo=false
f(x) = (x-2)* (x-2.5)*(x-3) / ((x-1)*(x+1))
p = plot(f, -20, -1-.3, legend=false, xlim=(-15, 15), color=:blue)
plot!(p, f, -1 + .2, 1 - .02, color=:blue)
plot!(p, f, 1 + .05, 20, color=:blue)
```
What kind of *asymptotes* does it appear to have?
```julia; hold=true; echo=false
choices = [
L"Just a horizontal asymptote, $y=0$",
L"Just vertical asymptotes at $x=-1$ and $x=1$",
L"Vertical asymptotes at $x=-1$ and $x=1$ and a horizontal asymptote $y=1$",
L"Vertical asymptotes at $x=-1$ and $x=1$ and a slant asymptote"
]
ans = 4
radioq(choices, ans)
```
###### Question
Consider the function ``p(x) = x + 2x^3 + 3x^3 + 4x^4 + 5x^5 +6x^6``. Which interval shows more than a ``U``-shaped graph that dominates for large ``x`` due to the leading term being ``6x^6``?
(Find an interval that contains the zeros, critical points, and inflection points.)
```julia; hold=true; echo=false
choices = ["``(-5,5)``, the default bounds of a calculator",
"``(-3.5, 3.5)``, the bounds given by Cauchy for the real roots of ``p``",
"``(-1, 1)``, as many special polynomials have their roots in this interval",
"``(-1.1, .25)``, as this constains all the roots, the critical points, and inflection points and just a bit more"
]
radioq(choices, 4, keep_order=true)
```
###### Question
Let ``f(x) = x^3/(9-x^2)``.
What points are *not* in the domain of ``f``?
```julia; echo=false
qchoices = [
"The values of `find_zeros(f, -10, 10)`: `[-3, 0, 3]`",
"The values of `find_zeros(f', -10, 10)`: `[-5.19615, 0, 5.19615]`",
"The values of `find_zeros(f'', -10, 10)`: `[-3, 0, 3]`",
"The zeros of the numerator: `[0]`",
"The zeros of the denominator: `[-3, 3]`",
"The value of `f(0)`: `0`",
"None of these choices"
]
radioq(qchoices, 5, keep_order=true)
```
The ``x``-intercepts are:
```julia; hold=true; echo=false
radioq(qchoices, 4, keep_order=true)
```
The ``y``-intercept is:
```julia; hold=true; echo=false
radioq(qchoices, 6, keep_order=true)
```
There are *vertical asymptotes* at ``x=\dots``?
```julia; hold=true; echo=false
radioq(qchoices, 5)
```
The *slant* asymptote has slope?
```julia; hold=true; echo=false
numericq(1)
```
The function has critical points at
```julia; hold=true,echo=false
radioq(qchoices, 2, keep_order=true)
```
The function has relative extrema at
```julia; hold=true;echo=false
radioq(qchoices, 7, keep_order=true)
```
The function has inflection points at
```julia; hold=true;echo=false
radioq(qchoices, 7, keep_order=true)
```
###### Question
A function ``f`` has
* zeros of ``\{-0.7548\dots, 2.0\}``,
* critical points at ``\{-0.17539\dots, 1.0, 1.42539\dots\}``,
* inflection points at ``\{0.2712\dots,1.2287\}``.
Is this a possible graph of ``f``?
```julia; hold=true;echo=false
f(x) = x^4 - 3x^3 + 2x^2 + x - 2
plot(f, -1, 2.5, legend=false)
```
```julia; hold=true;echo=false
yesnoq("yes")
```
###### Question
Two models for population growth are *exponential* growth: $P(t) = P_0 a^t$ and
[logistic growth](https://en.wikipedia.org/wiki/Logistic_function#In_ecology:_modeling_population_growth): $P(t) = K P_0 a^t / (K + P_0(a^t - 1))$. The exponential growth model has growth rate proportional to the current population. The logistic model has growth rate depending on the current population *and* the available resources (which can limit growth).
Letting $K=10$, $P_0=5$, and $a= e^{1/4}$. A plot over $[0,5]$ shows somewhat similar behaviour:
```julia;
K, P0, a = 50, 5, exp(1/4)
exponential_growth(t) = P0 * a^t
logistic_growth(t) = K * P0 * a^t / (K + P0*(a^t-1))
plot(exponential_growth, 0, 5)
plot!(logistic_growth)
```
Does a plot over $[0,50]$ show qualitatively similar behaviour?
```julia; hold=true; echo=false
yesnoq(true)
```
Exponential growth has $P''(t) = P_0 a^t \log(a)^2 > 0$, so has no inflection point. By plotting over a sufficiently wide interval, can you answer: does the logistic growth model have an inflection point?
```julia; hold=true; echo=false
yesnoq(true)
```
If yes, find it numerically:
```julia; hold=true; echo=false
val = find_zero(D(logistic_growth,2), (0, 20))
numericq(val)
```
The available resources are quantified by $K$. As $K \rightarrow \infty$ what is the limit of the logistic growth model:
```julia; hold=true; echo=false
choices = [
"The exponential growth model",
"The limit does not exist",
"The limit is ``P_0``"]
ans = 1
radioq(choices, ans)
```
##### Question
The plotting algorithm for plotting functions starts with a small
initial set of points over the specified interval ($21$) and then
refines those sub-intervals where the second derivative is determined
to be large.
Why are sub-intervals where the second derivative is large different than those where the second derivative is small?
```julia; hold=true; echo=false
choices = [
"The function will increase (or decrease) rapidly when the second derivative is large, so there needs to be more points to capture the shape",
"The function will have more curvature when the second derivative is large, so there needs to be more points to capture the shape",
"The function will be much larger (in absolute value) when the second derivative is large, so there needs to be more points to capture the shape",
]
ans = 2
radioq(choices, ans)
```
##### Question
Is there a nice algorithm to identify what domain a function should be
plotted over to produce an informative graph?
[Wilkinson](https://www.cs.uic.edu/~wilkinson/Publications/plotfunc.pdf)
has some suggestions. (Wilkinson is well known to the `R` community as
the specifier of the grammar of graphics.) It is mentioned that
"finding an informative domain for a given function depends on at least
three features: periodicity, asymptotics, and monotonicity."
Why would periodicity matter?
```julia; hold=true; echo=false
choices = [
"An informative graph only needs to show one or two periods, as others can be inferred.",
"An informative graph need only show a part of the period, as the rest can be inferred.",
L"An informative graph needs to show several periods, as that will allow proper computation for the $y$ axis range."]
ans = 1
radioq(choices, ans)
```
Why should asymptotics matter?
```julia; hold=true; echo=false
choices = [
L"A vertical asymptote can distory the $y$ range, so it is important to avoid too-large values",
L"A horizontal asymptote must be plotted from $-\infty$ to $\infty$",
"A slant asymptote must be plotted over a very wide domain so that it can be identified."
]
ans = 1
radioq(choices, ans)
```
Monotonicity means increasing or decreasing. This is important for what reason?
```julia; hold=true; echo=false
choices = [
"For monotonic regions, a large slope or very concave function might require more care to plot",
"For monotonic regions, a function is basically a straight line",
"For monotonic regions, the function will have a vertical asymptote, so the region should not be plotted"
]
ans = 1
radioq(choices, ans)
```

1512
CwJ/derivatives/derivatives.jmd Normal file
View File

File diff suppressed because it is too large Load Diff

BIN
CwJ/derivatives/figures/extrema-ladder.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

BIN
CwJ/derivatives/figures/extrema-rectangles.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

17
CwJ/derivatives/figures/extrema-ring-string.R Normal file
View File

@@ -0,0 +1,17 @@
## Used to make ring figure. Redo in Julia??
plot.new()
plot.window(xlim=c(0,1), ylim=c(-5, 1.1))
x <- seq(.1, .9, length=9)
y <- c(-4.46262,-4.46866, -4.47268, -4.47469, -4.47468, -4.47267, -4.46864, -4.4626 , -4.45454)
lines(c(0, x[3], 1), c(0, y[3], 1))
points(c(0,1), c(0,1), pch=16, cex=2)
text(c(0,1), c(0,1), c("(0,0)", c("(a,b)")), pos=3)
lines(c(0, x[3], x[3]), c(0, 0, y[3]), cex=2, col="gray")
lines(c(1, x[3], x[3]), c(1, 1, y[3]), cex=2, col="gray")
text(x[3]/2, 0, "x", pos=1)
text(x[3], y[3]/2, "|y|", pos=2)
text(x[3], (1 + y[3])/2, "b-y", pos=4)
text((x[3] + 1)/2, 1, "a-x", pos=1)
text(x[3], y[3], "0", cex=4, col="gold")

BIN
CwJ/derivatives/figures/extrema-ring-string.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

BIN
CwJ/derivatives/figures/fcarc-may2016-fig35-350.gif Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

BIN
CwJ/derivatives/figures/fcarc-may2016-fig40-300.gif Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 43 KiB

BIN
CwJ/derivatives/figures/fcarc-may2016-fig43-250.gif Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 44 KiB

BIN
CwJ/derivatives/figures/lhopital-32.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

BIN
CwJ/derivatives/figures/long-shadow-noir.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 221 KiB

986
CwJ/derivatives/first_second_derivatives.jmd Normal file
View File

@@ -0,0 +1,986 @@
# The first and second derivatives
This section uses these add-on packages:
```julia
using CalculusWithJulia
using Plots
using SymPy
using Roots
```
```julia; echo=false; results="hidden"
using CalculusWithJulia.WeaveSupport
const frontmatter = (
title = "The first and second derivatives",
description = "Calculus with Julia: The first and second derivatives",
tags = ["CalculusWithJulia", "derivatives", "the first and second derivatives"],
);
nothing
```
----
This section explores properties of a function, ``f(x)``, that are described by properties of its first and second derivatives, ``f'(x)`` and ``f''(x)``. As part of the conversation two tests are discussed that characterize when a critical point is a relative maximum or minimum. (We know that any relative maximum or minimum occurs at a critical point, but it is not true that *any* critical point will be a relative maximum or minimum.)
## Positive or increasing on an interval
We start with some vocabulary:
> A function $f$ is **positive** on an interval $I$ if for any $a$ in $I$ it must be that $f(a) > 0$.
Of course, we define *negative* in a parallel manner. The intermediate value theorem says a continuous function can not change from positive to negative without crossing $0$. This is not the case for functions with jumps, of course.
Next,
> A function, $f$, is (strictly) **increasing** on an interval $I$ if for any $a < b$ it must be that $f(a) < f(b)$.
The word strictly is related to the inclusion of the $<$ precluding the possibility of a function being flat over an interval that the $\leq$ inequality would allow.
A parallel definition with $a < b$ implying $f(a) > f(b)$ would be used for a *strictly decreasing* function.
We can try and prove these properties for a function algebraically --
we'll see both are related to the zeros of some function. However,
before proceeding to that it is usually helpful to get an idea of
where the answer is using exploratory graphs.
We will use a helper function, `plotif(f, g, a, b)` that plots the function `f` over `[a,b]` coloring it red when `g` is positive (and blue otherwise).
Such a function is defined for us in the accompanying `CalculusWithJulia` package, which has been previously been loaded.
To see where a function is positive, we simply pass the function
object in for *both* `f` and `g` above. For example, let's look at
where $f(x) = \sin(x)$ is positive:
```julia; hold=true;
f(x) = sin(x)
plotif(f, f, -2pi, 2pi)
```
Let's graph with `cos` in the masking spot and see what happens:
```julia;
plotif(sin, cos, -2pi, 2pi)
```
Maybe surprisingly, we see that the increasing parts of the sine curve are now
highlighted. Of course, the cosine is the derivative of the sine
function, now we discuss that this is no coincidence.
For the sequel, we will use `f'` notation to find numeric derivatives, with the notation being defined in the `CalculusWithJulia` package using the `ForwardDiff` package.
## The relationship of the derivative and increasing
The derivative, $f'(x)$, computes the slope of the tangent line to the
graph of $f(x)$ at the point $(x,f(x))$. If the derivative is
positive, the tangent line will have an increasing slope. Clearly if
we see an increasing function and mentally layer on a tangent line, it will
have a positive slope. Intuitively then, increasing functions and
positive derivatives are related concepts. But there are some
technicalities.
Suppose $f(x)$ has a derivative on $I$ . Then
> If $f'(x)$ is positive on an interval $I=(a,b)$, then $f(x)$ is strictly increasing on $I$.
Meanwhile,
> If a function $f(x)$ is increasing on $I$, then $f'(x) \geq 0$.
The technicality being the equality parts. In the second statement, we
have the derivative is non-negative, as we can't guarantee it is
positive, even if we considered just strictly increasing functions.
We can see by the example of $f(x) = x^3$ that strictly increasing
functions can have a zero derivative, at a point.
The mean value theorem provides the reasoning behind the first statement: on
$I$, the slope of any secant line between $d < e$ (both in $I$) is matched by the slope of some
tangent line, which by assumption will always be positive. If the
secant line slope is written as $(f(e) - f(d))/(e - d)$ with $d < e$,
then it is clear then that $f(e) - f(d) > 0$, or $d < e$ implies $f(d) < f(e)$.
The second part, follows from the secant line equation. The derivative
can be written as a limit of secant-line slopes, each of which is
positive. The limit of positive things can only be non-negative,
though there is no guarantee the limit will be positive.
So, to visualize where a function is increasing, we can just pass in
the derivative as the masking function in our `plotif` function, as long as we are wary about places with $0$ derivative (flat spots).
For example, here, with a more complicated function, the intervals where the function is
increasing are highlighted by passing in the functions derivative to `plotif`:
```julia; hold=true;
f(x) = sin(pi*x) * (x^3 - 4x^2 + 2)
plotif(f, f', -2, 2)
```
### First derivative test
When a function changes from increasing to decreasing, or decreasing to increasing, it will have a peak or a valley. More formally, such points are relative extrema.
When discussing the mean value thereom, we defined *relative
extrema* :
> * The function $f(x)$ has a *relative maximum* at $c$ if the value $f(c)$ is an *absolute maximum* for some *open* interval containing $c$.
> * Similarly, ``f(x)`` has a *relative minimum* at ``c`` if the value ``f(c)`` is an absolute minimum for *some* open interval about ``c``.
We know since [Fermat](http://tinyurl.com/nfgz8fz) that:
> Relative maxima and minima *must* occur at *critical* points.
Fermat says that *critical points* -- where the function is defined, but its derivative is either ``0`` or undefined -- are *interesting* points, however:
> A critical point need not indicate a relative maxima or minima.
Again, $f(x)=x^3$ provides the example at $x=0$. This is a critical point, but clearly not a
relative maximum or minimum - it is just a slight pause for a
strictly increasing function.
This leaves the question:
> When will a critical point correspond to a relative maximum or minimum?
This question can be answered by considering the first derivative.
> *The first derivative test*: If $c$ is a critical point for $f(x)$ and
> *if* $f'(x)$ changes sign at $x=c$, then $f(c)$ will be either a
> relative maximum or a relative minimum.
> * It will be a relative maximum if the derivative changes sign from $+$ to $-$.
> * It will be a relative minimum if the derivative changes sign from $-$ to $+$.
> * If $f'(x)$ does not change sign at $c$, then $f(c)$ is *not* a relative maximum or minimum.
The classification part, should be clear: e.g., if the derivative is positive then
negative, the function $f$ will increase to $(c,f(c))$ then decrease
from $(c,f(c))$ -- so ``f`` will have a local maximum at ``c``.
Our definition of critical point *assumes* $f(c)$ exists, as $c$ is in
the domain of $f$. With this assumption, vertical asymptotes are
avoided. However, it need not be that $f'(c)$ exists. The absolute
value function at $x=0$ provides an example: this point is a critical
point where the derivative changes sign, but ``f'(x)`` is not defined at exactly
$x=0$. Regardless, it is guaranteed that $f(c)$ will be a relative
minimum by the first derivative test.
##### Example
Consider the function $f(x) = e^{-\lvert x\rvert} \cos(\pi x)$ over $[-3,3]$:
```julia;
𝐟(x) = exp(-abs(x)) * cos(pi * x)
plotif(𝐟, 𝐟', -3, 3)
```
We can see the first derivative test in action: at the peaks and
valleys -- the relative extrema -- the color changes. This is because ``f'`` is changing sign as as the function
changes from increasing to decreasing or vice versa.
This function has a critical point at ``0``, as can be seen. It corresponds to a point where the derivative does not exist. It is still identified through `find_zeros`, which picks up zeros and in case of discontinuous functions, like `f'`, zero crossings:
```julia
find_zeros(𝐟', -3, 3)
```
##### Example
Find all the relative maxima and minima of the function $f(x) =
\sin(\pi \cdot x) \cdot (x^3 - 4x^2 + 2)$ over the interval $[-2, 2]$.
We will do so numerically. For
this task we first need to gather the critical points. As each of the
pieces of $f$ are everywhere differentiable and no quotients are
involved, the function $f$ will be everywhere differentiable. As such,
only zeros of $f'(x)$ can be critical points. We find these with
```julia;
𝒇(x) = sin(pi*x) * (x^3 - 4x^2 + 2)
𝒇cps = find_zeros(𝒇', -2, 2)
```
We should be careful though, as `find_zeros` may miss zeros that are not
simple or too close together. A critical point will correspond to a
relative maximum if the function crosses the axis, so these can not be
"pauses." As this is exactly the case we are screening for, we double
check that all the critical points are accounted for by graphing the
derivative:
```julia;
plot(𝒇', -2, 2, legend=false)
plot!(zero)
scatter!(𝒇cps, 0*𝒇cps)
```
We see the six zeros as stored in `cps` and note that at each the
function clearly crosses the $x$ axis.
From this last graph of the derivative we can also characterize the
graph of $f$: The left-most critical point coincides with a relative minimum
of $f$, as the derivative changes sign from negative to
positive. The critical points then alternate relative maximum,
relative minimum, relative maximum, relative, minimum, and finally relative maximum.
##### Example
Consider the function $g(x) = \sqrt{\lvert x^2 - 1\rvert}$. Find the critical
points and characterize them as relative extrema or not.
We will apply the same approach, but need to get a handle on how large
the values can be. The function is a composition of three
functions. We should expect that the only critical points will occur
when the interior polynomial, $x^2-1$ has values of interest, which is
around the interval $(-1, 1)$. So we look to the slightly wider interval $[-2, 2]$:
```julia;
g(x) = sqrt(abs(x^2 - 1))
gcps = find_zeros(g', -2, 2)
```
We see the three values $-1$, $0$, $1$ that correspond to the two
zeros and the relative minimum of $x^2 - 1$. We could graph things,
but instead we characterize these values using a sign chart. A
piecewise continuous function can only change sign when it crosses $0$ or jumps over ``0``. The
derivative will be continuous, except possibly at the three values
above, so is piecewise continuous.
A sign chart picks convenient values between crossing points to test if the function is positive or negative over those intervals. When computing by hand, these would ideally be values for which the function is easily computed. On the computer, this isn't a concern; below the midpoint is chosen:
```julia;
pts = sort(union(-2, gcps, 2)) # this includes the endpoints (a, b) and the critical points
test_pts = pts[1:end-1] + diff(pts)/2 # midpoints of intervals between pts
[test_pts sign.(g'.(test_pts))]
```
Such values are often summarized graphically on a number line using a *sign chart*:
```julia; eval=false
- ∞ + 0 - ∞ + g'
<---- -1 ----- 0 ----- 1 ---->
```
(The values where the function is ``0`` or could jump over ``0`` are shown on the number line, and the sign between these points is indicated. So the first minus sign shows ``g'(x)`` is *negative* on ``(-\infty, -1)``, the second minus sign shows ``g'(x)`` is negative on ``(0,1)``.)
Reading this we have:
- the derivative changes sign from negative to postive at $x=-1$, so $g(x)$ will have a relative minimum.
- the derivative changes sign from positive to negative at $x=0$, so $g(x)$ will have a relative maximum.
- the derivative changes sign from negative to postive at $x=1$, so $g(x)$ will have a relative minimum.
In the `CalculusWithJulia` package there is `sign_chart` function that will do such work for us, though with a different display:
```julia
sign_chart(g', -2, 2)
```
(This function numerically identifies ``x``-values for the specified function which are zeros, infinities, or points where the function jumps ``0``. It then shows the resulting sign pattern of the function from left to right.)
We did this all without graphs. But, let's look at the graph of the derivative:
```julia;
plot(g', -2, 2)
```
We see asymptotes at $x=-1$ and $x=1$! These aren't zeroes of $f'(x)$,
but rather where $f'(x)$ does not exist. The conclusion is correct -
each of $-1$, $0$ and $1$ are critical points with the identified characterization - but not for the
reason that they are all zeros.
```julia;
plot(g, -2, 2)
```
Finally, why does `find_zeros` find these values that are not zeros of
$g'(x)$? As discussed briefly above, it uses the bisection algorithm
on bracketing intervals to find zeros which are guaranteed by the
intermediate value theorem, but when applied to discontinuous functions, as `f'` is, will also identify values where the function jumps over ``0``.
##### Example
Consider the function $f(x) = \sin(x) - x$. Characterize the critical points.
We will work symbolically for this example.
```julia;
@syms x
fx = sin(x) - x
fp = diff(fx, x)
solve(fp)
```
We get values of $0$ and $2\pi$. Let's look at the derivative at these points:
At $x=0$ we have to the left and right signs found by
```julia;
fp(-pi/2), fp(pi/2)
```
Both are negative. The derivative does not change sign at $0$, so the critical point is neither a relative minimum or maximum.
What about at $2\pi$? We do something similar:
```julia;
fp(2pi - pi/2), fp(2pi + pi/2)
```
Again, both negative. The function $f(x)$ is just decreasing near
$2\pi$, so again the critical point is neither a relative minimum or maximum.
A graph verifies this:
```julia;
plot(fx, -3pi, 3pi)
```
We see that at $0$ and $2\pi$ there are "pauses" as the function
decreases. We should also see that this pattern repeats. The critical
points found by `solve` are only those within a certain domain. Any
value that satisfies $\cos(x) - 1 = 0$ will be a critical point, and
there are infinitely many of these of the form $n \cdot 2\pi$ for $n$
an integer.
As a comment, the `solveset` function, which is replacing `solve`,
returns the entire collection of zeros:
```julia;
solveset(fp)
```
----
Of course, `sign_chart` also does this, only numerically. We just need to pick an interval wide enough to contains ``[0,2\pi]``
```julia
sign_chart((x -> sin(x)-x)', -3pi, 3pi)
```
##### Example
Suppose you know $f'(x) = (x-1)\cdot(x-2)\cdot (x-3) = x^3 - 6x^2 +
11x - 6$ and $g'(x) = (x-1)\cdot(x-2)^2\cdot(x-3)^3 = x^6 -14x^5
+80x^4-238x^3+387x^2-324x+108$.
How would the graphs of $f(x)$ and $g(x)$ differ, as they share identical critical points?
The graph of $f(x)$ - a function we do not have a formula for - can have its critical points characterized by the first derivative test. As the derivative changes sign at each, all critical points correspond to relative maxima. The sign pattern is negative/positive/negative/positive so we have from left to right a relative minimum, a relative maximum, and then a relative minimum. This is consistent with a ``4``th degree polynomial with ``3`` relative extrema.
For the graph of $g(x)$ we can apply the same analysis. Thinking for a
moment, we see as the factor $(x-2)^2$ comes as a power of $2$, the
derivative of $g(x)$ will not change sign at $x=2$, so there is no
relative extreme value there. However, at $x=3$ the factor has an odd
power, so the derivative will change sign at $x=3$. So, as $g'(x)$ is
positive for large *negative* values, there will be a relative maximum
at $x=1$ and, as $g'(x)$ is positive for large *positive* values, a
relative minimum at $x=3$.
The latter is consistent with a $7$th degree polynomial with positive leading coefficient. It is intuitive that since $g'(x)$ is a $6$th degree polynomial, $g(x)$ will be a $7$th degree one, as the power rule applied to a polynomial results in a polynomial of lesser degree by one.
Here is a simple schematic that illustrates the above considerations.
```julia; eval=false
f' - 0 + 0 - 0 + f'-sign
↘ ↗ ↘ ↗ f-direction
∪ ∩ ∪ f-shape
g' + 0 - 0 - 0 + g'-sign
↗ ↘ ↘ ↗ g-direction
∩ ~ ∪ g-shape
<------ 1 ----- 2 ----- 3 ------>
```
## Concavity
Consider the function $f(x) = x^2$. Over this function we draw some
secant lines for a few pairs of $x$ values:
```julia; hold=true; echo=false
f(x) = x^2
seca(f,a,b) = x -> f(a) + (f(b) - f(a)) / (b-a) * (x-a)
p = plot(f, -2, 3, legend=false, linewidth=5, xlim=(-2,3), ylim=(-2, 9))
plot!(p,seca(f, -1, 2))
a,b = -1, 2; xs = range(a, stop=b, length=50)
plot!(xs, seca(f, a, b).(xs), linewidth=5)
plot!(p,seca(f, 0, 3/2))
a,b = 0, 3/2; xs = range(a, stop=b, length=50)
plot!(xs, seca(f, a, b).(xs), linewidth=5)
p
```
The graph attempts to illustrate that for this function the secant
line between any two points $a < b$ will lie above the graph over $[a,b]$.
This is a special property not shared by all functions. Let $I$ be an open interval.
> **Concave up**: A function $f(x)$ is concave up on $I$ if for any $a < b$ in $I$, the secant line between $a$ and $b$ lies above the graph of $f(x)$ over $[a,b]$.
A similar definition exists for *concave down* where the secant lines
lie below the graph. Notationally, concave up says for any $x$ in $[a,b]$:
```math
f(a) + \frac{f(b) - f(a)}{b-a} \cdot (x-a) \geq f(x) \quad\text{ (concave up) }
```
Replacing
$\geq$ with $\leq$ defines *concave down*, and with either $>$ or $<$
will add the prefix "strictly." These definitions are useful for a
general definition of
[convex functions](https://en.wikipedia.org/wiki/Convex_function).
We won't work with these definitions in this section, rather we will characterize
concavity for functions which have either a first or second
derivative:
> * If $f'(x)$ exists and is *increasing* on $(a,b)$, then $f(x)$ is concave up on $(a,b)$.
> * If ``f'(x)`` is *decreasing* on ``(a,b)``, then ``f(x)`` is concave *down*.
A proof of this makes use of the same trick used to establish the mean
value theorem from Rolle's theorem. Assume ``f'`` is increasing and let
$g(x) = f(x) - (f(a) + M \cdot (x-a))$, where $M$ is the slope of
the secant line between $a$ and $b$. By construction $g(a) = g(b) =
0$. If $f'(x)$ is increasing, then so is $g'(x) = f'(x) + M$. By its
definition above, showing ``f`` is concave up is the same as showing $g(x) \leq
0$. Suppose to the contrary that there is a value where $g(x) > 0$ in
$[a,b]$. We show this can't be. Assuming $g'(x)$ always exists, after
some work, Rolle's theorem will ensure there is a value where $g'(c) =
0$ and $(c,g(c))$ is a relative maximum, and as we know there is at
least one positive value, it must be $g(c) > 0$. The first derivative
test then ensures that $g'(x)$ will increase to the left of $c$ and
decrease to the right of $c$, since $c$ is at a critical point and not
an endpoint. But this can't happen as $g'(x)$ is assumed to be
increasing on the interval.
The relationship between increasing functions and their derivatives -- if $f'(x) > 0 $ on $I$, then ``f`` is increasing on $I$ --
gives this second characterization of concavity when the second
derivative exists:
> * If $f''(x)$ exists and is positive on $I$, then $f(x)$ is concave up on $I$.
> * If $f''(x)$ exists and is negative on $I$, then $f(x)$ is concave down on $I$.
This follows, as we can think of $f''(x)$ as just the first derivative
of the function $f'(x)$, so the assumption will force $f'(x)$ to exist and be
increasing, and hence $f(x)$ to be concave up.
##### Example
Let's look at the function $x^2 \cdot e^{-x}$ for positive $x$. A
quick graph shows the function is concave up, then down, then up in
the region plotted:
```julia;
h(x) = x^2 * exp(-x)
plotif(h, h'', 0, 8)
```
From the graph, we would expect that the second derivative - which is continuous - would have two zeros on $[0,8]$:
```julia;
ips = find_zeros(h'', 0, 8)
```
As well, between the zeros we should have the sign pattern `+`, `-`, and `+`, as we verify:
```julia;
sign_chart(h'', 0, 8)
```
### Second derivative test
Concave up functions are "opening" up, and often clearly $U$-shaped, though that is not necessary. At a
relative minimum, where there is a ``U``-shape, the graph will be concave up; conversely
at a relative maximum, where the graph has a downward ``\cap``-shape, the function will be concave down. This observation becomes:
> The **second derivative test**: If $c$ is a critical point of $f(x)$
> with $f''(c)$ existing in a neighborhood of $c$, then
> * The value $f(c)$ will be a relative maximum if $f''(c) > 0$,
> * The value $f(c)$ will be a relative minimum if $f''(c) < 0$, and
> * *if* ``f''(c) = 0`` the test is *inconclusive*.
If $f''(c)$ is positive in an interval about $c$, then $f''(c) > 0$ implies the function is
concave up at $x=c$. In turn, concave up implies the derivative is increasing
so must go from negative to positive at the critical point.
The second derivative test is **inconclusive** when $f''(c)=0$. No such
general statement exists, as there isn't enough information. For
example, the function $f(x) = x^3$ has $0$ as a critical point,
$f''(0)=0$ and the value does not correspond to a relative maximum or minimum. On the
other hand $f(x)=x^4$ has $0$ as a critical point, $f''(0)=0$ is a
relative minimum.
##### Example
Use the second derivative test to characterize the critical points of $j(x) = x^5 - x^4 + x^3$.
```julia;
j(x) = x^5 - 2x^4 + x^3
jcps = find_zeros(j', -3, 3)
```
We can check the sign of the second derivative for each critical point:
```julia;
[jcps j''.(jcps)]
```
That $j''(0.6) < 0$ implies that at $0.6$, $j(x)$ will have a relative
maximum. As $''(1) > 0$, the second derivative test says at $x=1$
there will be a relative minimum. That $j''(0) = 0$ says that only
that there **may** be a relative maximum or minimum at $x=0$, as the second
derivative test does not speak to this situation. (This last check, requiring a function evaluation to be `0`, is susceptible to floating point errors, so isn't very robust as a general tool.)
This should be consistent with
this graph, where $-0.25$, and $1.25$ are chosen to capture the zero at
$0$ and the two relative extrema:
```julia;
plotif(j, j'', -0.25, 1.25)
```
For the graph we see that $0$ **is not** a relative maximum or minimum. We could have seen this numerically by checking the first derivative test, and noting there is no sign change:
```julia;
sign_chart(j', -3, 3)
```
##### Example
One way to visualize the second derivative test is to *locally* overlay on a critical point a parabola. For example, consider ``f(x) = \sin(x) + \sin(2x) + \sin(3x)`` over ``[0,2\pi]``. It has ``6`` critical points over ``[0,2\pi]``. In this graphic, we *locally* layer on ``6`` parabolas:
```julia; hold=true;
f(x) = sin(x) + sin(2x) + sin(3x)
p = plot(f, 0, 2pi, legend=false, color=:blue, linewidth=3)
cps = fzeros(f', 0, 2pi)
h = 0.5
for c in cps
parabola(x) = f(c) + (f''(c)/2) * (x-c)^2
plot!(parabola, c-h, c+h, color=:red, linewidth=5, alpha=0.6)
end
p
```
The graphic shows that for this function near the relative extrema the parabolas *approximate* the function well, so that the relative extrema are characterized by the relative extrema of the parabolas.
At each critical point ``c``, the parabolas have the form
```math
f(c) + \frac{f''(c)}{2}(x-c)^2.
```
The ``2`` is a mystery to be answered in the section on [Taylor series](../taylor_series_polynomials.html), the focus here is on the *sign* of ``f''(c)``:
* if ``f''(c) > 0`` then the approximating parabola opens upward and the critical point is a point of relative minimum for ``f``,
* if ``f''(c) < 0`` then the approximating parabola opens downward and the critical point is a point of relative maximum for ``f``, and
* were ``f''(c) = 0`` then the approximating parabola is just a line -- the tangent line at a critical point -- and is non-informative about extrema.
That is, the parabola picture is just the second derivative test in this light.
### Inflection points
An inflection point is a value where the *second* derivative of $f$
changes sign. At an inflection point the derivative will change from
increasing to decreasing (or vice versa) and the function will change
from concave up to down (or vice versa).
We can use the `find_zeros` function to identify potential inflection
points by passing in the second derivative function. For example,
consider the bell-shaped function
```math
k(x) = e^{-x^2/2}.
```
A graph suggests relative a maximum at $x=0$, a horizontal asymptote of $y=0$,
and two inflection points:
```julia;
k(x) = exp(-x^2/2)
plotif(k, k'', -3, 3)
```
The inflection points can be found directly, if desired, or numerically with:
```julia;
find_zeros(k'', -3, 3)
```
(The `find_zeros` function may return points which are not inflection points. It primarily returns points where $k''(x)$ changes sign, but *may* also find points where $k''(x)$ is $0$ yet does not change sign at $x$.)
##### Example
A car travels from a stop for 1 mile in 2 minutes. A graph of its
position as a function of time might look like any of these graphs:
```julia; hold=true; echo=false
v(t) = 30/60*t
w(t) = t < 1/2 ? 0.0 : (t > 3/2 ? 1.0 : (t-1/2))
y(t) = 1 / (1 + exp(-t))
y1(t) = y(2(t-1))
y2(t) = y1(t) - y1(0)
y3(t) = 1/y2(2) * y2(t)
plot(v, 0, 2, label="f1")
plot!(w, label="f2")
plot!(y3, label="f3")
```
All three graphs have the same *average* velocity which is just the
$1/2$ miles per minute (``30`` miles an hour). But the instantaneous
velocity - which is given by the derivative of the position function)
varies.
The graph `f1` has constant velocity, so the position is a straight
line with slope $v_0$. The graph `f2` is similar, though for first and
last 30 seconds, the car does not move, so must move faster during the
time it moves. A more realistic graph would be `f3`. The position
increases continuously, as do the others, but the velocity changes
more gradually. The initial velocity is less than $v_0$, but
eventually gets to be more than $v_0$, then velocity starts to
increase less. At no point is the velocity not increasing, for `f3`,
the way it is for `f2` after a minute and a half.
The rate of change of the velocity is the acceleration. For `f1` this
is zero, for `f2` it is zero as well - when it is defined. However,
for `f3` we see the increase in velocity is positive in the first
minute, but negative in the second minute. This fact relates to the
concavity of the graph. As acceleration is the derivative of velocity,
it is the second derivative of position - the graph we see. Where the
acceleration is *positive*, the position graph will be concave *up*,
where the acceleration is *negative* the graph will be concave
*down*. The point $t=1$ is an inflection point, and
would be felt by most riders.
## Questions
###### Question
Consider this graph:
```julia;
plot(airyai, -5, 0) # airyai in `SpecialFunctions` loaded with `CalculusWithJulia`
```
On what intervals (roughly) is the function positive?
```julia; hold=true; echo=false
choices=[
"``(-3.2,-1)``",
"``(-5, -4.2)``",
"``(-5, -4.2)`` and ``(-2.5, 0)``",
"``(-4.2, -2.5)``"]
ans = 3
radioq(choices, ans)
```
###### Question
Consider this graph:
```julia; hold=true; echo=false
import SpecialFunctions: besselj
p = plot(x->besselj(x, 1), -5,-3)
```
On what intervals (roughly) is the function negative?
```julia; hold=true; echo=false
choices=[
"``(-5.0, -4.0)``",
"``(-25.0, 0.0)``",
"``(-5.0, -4.0)`` and ``(-4, -3)``",
"``(-4.0, -3.0)``"]
ans = 4
radioq(choices, ans)
```
###### Question
Consider this graph
```julia; hold=true; echo=false
plot(x->besselj(x, 21), -5,-3)
```
On what interval(s) is this function increasing?
```julia; hold=true; echo=false
choices=[
"``(-5.0, -3.8)``",
"``(-3.8, -3.0)``",
"``(-4.7, -3.0)``",
"``(-0.17, 0.17)``"
]
ans = 3
radioq(choices, ans)
```
###### Question
Consider this graph
```julia; hold=true; echo=false
p = plot(x -> 1 / (1+x^2), -3, 3)
```
On what interval(s) is this function concave up?
```julia; hold=true; echo=false
choices=[
"``(0.1, 1.0)``",
"``(-3.0, 3.0)``",
"``(-0.6, 0.6)``",
" ``(-3.0, -0.6)`` and ``(0.6, 3.0)``"
]
ans = 4
radioq(choices, ans)
```
###### Question
If it is known that:
* A function $f(x)$ has critical points at $x=-1, 0, 1$
* at $-2$ an $-1/2$ the values are: $f'(-2) = 1$ and $f'(-1/2) = -1$.
What can be concluded?
```julia; hold=true; echo=false
choices = [
"Nothing",
"That the critical point at ``-1`` is a relative maximum",
"That the critical point at ``-1`` is a relative minimum",
"That the critical point at ``0`` is a relative maximum",
"That the critical point at ``0`` is a relative minimum"
]
ans = 2
radioq(choices, ans, keep_order=true)
```
###### Question
Mystery function $f(x)$ has $f'(2) = 0$ and $f''(0) = 2$. What is the *most* you can say about $x=2$?
```julia; hold=true; echo=false
choices = [
" ``f(x)`` is continuous at ``2``",
" ``f(x)`` is continuous and differentiable at ``2``",
" ``f(x)`` is continuous and differentiable at ``2`` and has a critical point",
" ``f(x)`` is continuous and differentiable at ``2`` and has a critical point that is a relative minimum by the second derivative test"
]
ans = 3
radioq(choices, ans, keep_order=true)
```
###### Question
Find the smallest critical point of $f(x) = x^3 e^{-x}$.
```julia; hold=true; echo=false
f(x)= x^3*exp(-x)
cps = find_zeros(D(f), -5, 10)
val = minimum(cps)
numericq(val)
```
###### Question
How many critical points does $f(x) = x^5 - x + 1$ have?
```julia; hold=true; echo=false
f(x) = x^5 - x + 1
cps = find_zeros(D(f), -3, 3)
val = length(cps)
numericq(val)
```
###### Question
How many inflection points does $f(x) = x^5 - x + 1$ have?
```julia; hold=true; echo=false
f(x) = x^5 - x + 1
cps = find_zeros(D(f,2), -3, 3)
val = length(cps)
numericq(val)
```
###### Question
At $c$, $f'(c) = 0$ and $f''(c) = 1 + c^2$. Is $(c,f(c))$ a relative maximum? ($f$ is a "nice" function.)
```julia; hold=true; echo=false
choices = [
"No, it is a relative minimum",
"No, the second derivative test is possibly inconclusive",
"Yes"
]
ans = 1
radioq(choices, ans)
```
###### Question
At $c$, $f'(c) = 0$ and $f''(c) = c^2$. Is $(c,f(c))$ a relative minimum? ($f$ is a "nice" function.)
```julia; hold=true; echo=false
choices = [
"No, it is a relative maximum",
"No, the second derivative test is possibly inconclusive if ``c=0``, but otherwise yes",
"Yes"
]
ans = 2
radioq(choices, ans)
```
###### Question
```julia; hold=true; echo=false
f(x) = exp(-x) * sin(pi*x)
plot(D(f), 0, 3)
```
The graph shows $f'(x)$. Is it possible that $f(x) = e^{-x} \sin(\pi x)$?
```julia; hold=true; echo=false
yesnoq(true)
```
(Plot ``f(x)`` and compare features like critical points, increasing decreasing to that indicated by ``f'`` through the graph.)
###### Question
```julia; hold=true; echo=false
f(x) = x^4 - 3x^3 - 2x + 4
plot(D(f,2), -2, 4)
```
The graph shows $f'(x)$. Is it possible that $f(x) = x^4 - 3x^3 - 2x + 4$?
```julia; hold=true; echo=false
yesnoq("no")
```
###### Question
```julia; hold=true; echo=false
f(x) = (1+x)^(-2)
plot(D(f,2), 0,2)
```
The graph shows $f''(x)$. Is it possible that $f(x) = (1+x)^{-2}$?
```julia; hold=true; echo=false
yesnoq("yes")
```
###### Question
```julia; hold=true; echo=false
f_p(x) = (x-1)*(x-2)^2*(x-3)^2
plot(f_p, 0.75, 3.5)
```
This plot shows the graph of $f'(x)$. What is true about the critical points and their characterization?
```julia; hold=true; echo=false
choices = [
"The critical points are at ``x=1`` (a relative minimum), ``x=2`` (not a relative extrema), and ``x=3`` (not a relative extrema).",
"The critical points are at ``x=1`` (a relative maximum), ``x=2`` (not a relative extrema), and ``x=3`` (not a relative extrema).",
"The critical points are at ``x=1`` (a relative minimum), ``x=2`` (not a relative extrema), and ``x=3`` (a relative minimum).",
"The critical points are at ``x=1`` (a relative minimum), ``x=2`` (a relative minimum), and ``x=3`` (a relative minimum).",
]
ans=1
radioq(choices, ans)
```
##### Question
You know $f''(x) = (x-1)^3$. What do you know about $f(x)$?
```julia; hold=true; echo=false
choices = [
"The function is concave down over ``(-\\infty, 1)`` and concave up over ``(1, \\infty)``",
"The function is decreasing over ``(-\\infty, 1)`` and increasing over ``(1, \\infty)``",
"The function is negative over ``(-\\infty, 1)`` and positive over ``(1, \\infty)``",
]
ans = 1
radioq(choices, ans)
```
##### Question
While driving we accelerate to get through a light before it turns red. However, at time $t_0$ a car cuts in front of us and we are forced to break. If $s(t)$ represents position, what is $t_0$:
```julia; hold=true; echo=false
choices = ["A zero of the function",
"A critical point for the function",
"An inflection point for the function"]
ans = 3
radioq(choices, ans, keep_order=true)
```
###### Question
The [investopedia](https://www.investopedia.com/terms/i/inflectionpoint.asp) website describes:
"An **inflection point** is an event that results in a significant change in the progress of a company, industry, sector, economy, or geopolitical situation and can be considered a turning point after which a dramatic change, with either positive or negative results, is expected to result."
This accurately summarizes how the term is used outside of math books. Does it also describe how the term is used *inside* math books?
```julia; hold=true, echo=false
choices = ["Yes. Same words, same meaning",
"""No, but it is close. An inflection point is when the *acceleration* changes from positive to negative, so if "results" are about how a company's rate of change is changing, then it is in the ballpark."""]
radioq(choices, 2)
```
###### Question
The function ``f(x) = x^3 + x^4`` has a critical point at ``0`` and a second derivative of ``0`` at ``x=0``. Without resorting to the first derivative test, and only considering that *near* ``x=0`` the function ``f(x)`` is essentially ``x^3``, as ``f(x) = x^3(1+x)``, what can you say about whether the critical point is a relative extrema?
```julia; hold=true; echo=false
choices = ["As ``x^3`` has no extrema at ``x=0``, neither will ``f``",
"As ``x^4`` is of higher degree than ``x^3``, ``f`` will be ``U``-shaped, as ``x^4`` is."]
radioq(choices, 1)
```

1025
CwJ/derivatives/implicit_differentiation.jmd Normal file
View File

File diff suppressed because it is too large Load Diff

774
CwJ/derivatives/lhospitals_rule.jmd Normal file
View File

@@ -0,0 +1,774 @@
# L'Hospital's Rule
This section uses these add-on packages:
```julia
using CalculusWithJulia
using Plots
using SymPy
```
```julia; echo=false; results="hidden"
using CalculusWithJulia.WeaveSupport
using Roots
fig_size=(600, 400)
const frontmatter = (
title = "L'Hospital's Rule",
description = "Calculus with Julia: L'Hospital's Rule",
tags = ["CalculusWithJulia", "derivatives", "l'hospital's rule"],
);
nothing
```
----
Let's return to limits of the form ``\lim_{x \rightarrow c}f(x)/g(x)`` which have an
indeterminate form of ``0/0`` if both are evaluated at ``c``. The typical
example being the limit considered by Euler:
```math
\lim_{x\rightarrow 0} \frac{\sin(x)}{x}.
```
We know this is ``1`` using a bound from geometry, but might also guess
this is one, as we know from linearization near ``0`` that we have ``\sin(x) \approx x`` or, more specifically:
```math
\sin(x) = x - \sin(\xi)x^2/2, \quad 0 < \xi < x.
```
This would yield:
```math
\lim_{x \rightarrow 0} \frac{\sin(x)}{x} = \lim_{x\rightarrow 0} \frac{x -\sin(\xi) x^2/2}{x} = \lim_{x\rightarrow 0} 1 + \sin(\xi) \cdot x/2 = 1.
```
This is because we know ``\sin(\xi) x/2`` has a limit of ``0``, when ``|\xi| \leq |x|``.
That doesn't look any easier, as we worried about the error term, but
if just mentally replaced ``\sin(x)`` with ``x`` - which it basically is
near ``0`` - then we can see that the limit should be the same as ``x/x``
which we know is ``1`` without thinking.
Basically, we found that in terms of limits, if both ``f(x)`` and ``g(x)``
are ``0`` at ``c``, that we *might* be able to just take this limit:
``(f(c) + f'(c) \cdot(x-c)) / (g(c) + g'(c) \cdot (x-c))`` which is just
``f'(c)/g'(c)``.
Wouldn't that be nice? We could find difficult limits just by
differentiating the top and the bottom at ``c`` (and not use the messy quotient rule).
Well, in fact that is more or less true, a fact that dates back to
[L'Hospital](http://en.wikipedia.org/wiki/L%27H%C3%B4pital%27s_rule) -
who wrote the first textbook on differential calculus - though this result is
likely due to one of the Bernoulli brothers.
> *L'Hospital's rule*: Suppose:
> * that ``\lim_{x\rightarrow c+} f(c) =0`` and ``\lim_{x\rightarrow c+} g(c) =0``,
> * that ``f`` and ``g`` are differentiable in ``(c,b)``, and
> * that ``g(x)`` exists and is non-zero for *all* ``x`` in ``(c,b)``,
> then **if** the following limit exists:
> ``\lim_{x\rightarrow c+}f'(x)/g'(x)=L`` it follows that
> ``\lim_{x \rightarrow c+}f(x)/g(x) = L``.
That is *if* the right limit of ``f(x)/g(x)`` is indeterminate of the form ``0/0``,
but the right limit of ``f'(x)/g'(x)`` is known, possibly by simple
continuity, then the right limit of ``f(x)/g(x)`` exists and is equal to that
of ``f'(x)/g'(x)``.
The rule equally applies to *left limits* and *limits* at ``c``. Later it will see there are other generalizations.
To apply this rule to Euler's example, ``\sin(x)/x``, we just need to consider that:
```math
L = 1 = \lim_{x \rightarrow 0}\frac{\cos(x)}{1},
```
So, as well, ``\lim_{x \rightarrow 0} \sin(x)/x = 1``.
This is due to ``\cos(x)`` being continuous at ``0``, so this limit is
just ``\cos(0)/1``. (More importantly, the tangent line expansion of
``\sin(x)`` at ``0`` is ``\sin(0) + \cos(0)x``, so that ``\cos(0)`` is why
this answer is as it is, but we don't need to think in terms of
``\cos(0)``, but rather the tangent-line expansion, which is ``\sin(x)
\approx x``, as ``\cos(0)`` appears as the coefficient.
```julia; echo=false
note("""
In [Gruntz](http://www.cybertester.com/data/gruntz.pdf), in a reference attributed to Speiss, we learn that L'Hospital was a French Marquis who was taught in ``1692`` the calculus of Leibniz by Johann Bernoulli. They made a contract obliging Bernoulli to leave his mathematical inventions to L'Hospital in exchange for a regular compensation. This result was discovered in ``1694`` and appeared in L'Hospital's book of ``1696``.
"""; title="Bernoulli-de l'Hospital")
```
##### Examples
- Consider this limit at ``0``: ``(a^x - 1)/x``. We have ``f(x) =a^x-1`` has
``f(0) = 0``, so this limit is indeterminate of the form ``0/0``. The
derivative of ``f(x)`` is ``f'(x) = a^x \log(a)`` which has ``f'(0) = \log(a)``.
The derivative of the bottom is also ``1`` at ``0``, so we have:
```math
\log(a) = \frac{\log(a)}{1} = \frac{f'(0)}{g'(0)} = \lim_{x \rightarrow 0}\frac{f'(x)}{g'(x)} = \lim_{x \rightarrow 0}\frac{f(x)}{g(x)}
= \lim_{x \rightarrow 0}\frac{a^x - 1}{x}.
```
```julia; echo=false
note("""Why rewrite in the "opposite" direction? Because the theorem's result -- ``L`` is the limit -- is only true if the related limit involving the derivative exists. We don't do this in the following, but did so here to emphasize the need for the limit of the ratio of the derivatives to exist.
""")
```
- Consider this limit:
```math
\lim_{x \rightarrow 0} \frac{e^x - e^{-x}}{x}.
```
It too is of the indeterminate form ``0/0``. The derivative of the top
is ``e^x + e^{-x}``, which is ``2`` when ``x=0``, so the ratio of
``f'(0)/g'(0)`` is seen to be ``2`` By continuity, the limit of the ratio of the derivatives is ``2``. Then by L'Hospital's rule, the limit above is
``2``.
- Sometimes, L'Hospital's rule must be applied twice. Consider this
limit:
```math
\lim_{x \rightarrow 0} \frac{\cos(x)}{1 - x^2}
```
By L'Hospital's rule *if* this following limit exists, the two will be equal:
```math
\lim_{x \rightarrow 0} \frac{-\sin(x)}{-2x}.
```
But if we didn't guess the answer, we see that this new problem is *also* indeterminate
of the form ``0/0``. So, repeating the process, this new limit will exist and be equal to the following
limit, should it exist:
```math
\lim_{x \rightarrow 0} \frac{-\cos(x)}{-2} = 1/2.
```
As ``L = 1/2`` for this related limit, it must also be the limit of the original problem, by L'Hospital's rule.
- Our "intuitive" limits can bump into issues. Take for example the limit of ``(\sin(x)-x)/x^2`` as ``x`` goes to ``0``. Using ``\sin(x) \approx x`` makes this look like ``0/x^2`` which is still indeterminate. (Because the difference is higher order than ``x``.) Using L'Hospitals, says this limit will exist (and be equal) if the following one does:
```math
\lim_{x \rightarrow 0} \frac{\cos(x) - 1}{2x}.
```
This particular limit is indeterminate of the form ``0/0``, so we again try L'Hospital's rule and consider
```math
\lim_{x \rightarrow 0} \frac{-\sin(x)}{2} = 0
```
So as this limit exists, working backwards, the original limit in question will also be ``0``.
- This example comes from the Wikipedia page. It "proves" a discrete approximation for the second derivative.
Show if ``f''(x)`` exists at ``c`` and is continuous at ``c``, then
```math
f''(c) = \lim_{h \rightarrow 0} \frac{f(c + h) - 2f(c) + f(c-h)}{h^2}.
```
This will follow from two applications of L'Hospital's rule to the
right-hand side. The first says, the limit on the right is equal to
this limit, should it exist:
```math
\lim_{h \rightarrow 0} \frac{f'(c+h) - 0 - f'(c-h)}{2h}.
```
We have to be careful, as we differentiate in the ``h`` variable, not
the ``c`` one, so the chain rule brings out the minus sign. But again,
as we still have an indeterminate form ``0/0``, this limit will equal the
following limit should it exist:
```math
\lim_{h \rightarrow 0} \frac{f''(c+h) - 0 - (-f''(c-h))}{2} =
\lim_{c \rightarrow 0}\frac{f''(c+h) + f''(c-h)}{2} = f''(c).
```
That last equality follows, as it is assumed that ``f''(x)`` exists at ``c`` and is continuous, that is, ``f''(c \pm h) \rightarrow f''(c)``.
The expression above finds use when second derivatives are numerically approximated. (The middle expression is the basis of the central-finite difference approximation to the derivative.)
* L'Hospital himself was interested in this limit for ``a > 0`` ([math overflow](http://mathoverflow.net/questions/51685/how-did-bernoulli-prove-lh%C3%B4pitals-rule))
```math
\lim_{x \rightarrow a} \frac{\sqrt{2a^3\cdot x-x^4} - a\cdot(a^2\cdot x)^{1/3}}{ a - (a\cdot x^3)^{1/4}}.
```
These derivatives can be done by hand, but to avoid any minor mistakes
we utilize `SymPy` taking care to use rational numbers for the
fractional powers, so as not to lose precision through floating point
roundoff:
```julia;
@syms a::positive x::positive
f(x) = sqrt(2a^3*x - x^4) - a * (a^2*x)^(1//3)
g(x) = a - (a*x^3)^(1//4)
```
We can see that at ``x=a`` we have the indeterminate form ``0/0``:
```julia;
f(a), g(a)
```
What about the derivatives?
```julia;
fp, gp = diff(f(x),x), diff(g(x),x)
fp(x=>a), gp(x=>a)
```
Their ratio will not be indeterminate, so the limit in question is just the ratio:
```julia;
fp(x=>a) / gp(x=>a)
```
Of course, we could have just relied on `limit`, which knows about L'Hospital's rule:
```julia;
limit(f(x)/g(x), x, a)
```
## Idea behind L'Hospital's rule
A first proof of L'Hospital's rule takes advantage of Cauchy's
[generalization](http://en.wikipedia.org/wiki/Mean_value_theorem#Cauchy.27s_mean_value_theorem)
of the mean value theorem to two functions. Suppose ``f(x)`` and ``g(x)`` are
continuous on ``[c,b]`` and differentiable on ``(c,b)``. On
``(c,x)``, ``c < x < b`` there exists a ``\xi`` with ``f'(\xi) \cdot (f(x) - f(c)) =
g'(\xi) \cdot (g(x) - g(c))``. In our formulation, both ``f(c)`` and ``g(c)``
are zero, so we have, provided we know that ``g(x)`` is non zero, that
``f(x)/g(x) = f'(\xi)/g'(\xi)`` for some ``\xi``, ``c < \xi < c + x``. That
the right-hand side has a limit as ``x \rightarrow c+`` is true by the
assumption that the limit of the ratio of the derivatives exists. (The ``\xi``
part can be removed by considering it as a composition of a function
going to ``c``.) Thus the right limit of the ratio ``f/g`` is
known.
----
```julia; hold=true; echo=false; cache=true
## {{{lhopitals_picture}}}
function lhopitals_picture_graph(n)
g = (x) -> sqrt(1 + x) - 1 - x^2
f = (x) -> x^2
ts = range(-1/2, stop=1/2, length=50)
a, b = 0, 1/2^n * 1/2
m = (f(b)-f(a)) / (g(b)-g(a))
## get bounds
tl = (x) -> g(0) + m * (x - f(0))
lx = max(fzero(x -> tl(x) - (-0.05),-1000, 1000), -0.6)
rx = min(fzero(x -> tl(x) - (0.25),-1000, 1000), 0.2)
xs = [lx, rx]
ys = map(tl, xs)
plt = plot(g, f, -1/2, 1/2, legend=false, size=fig_size, xlim=(-.6, .5), ylim=(-.1, .3))
plot!(plt, xs, ys, color=:orange)
scatter!(plt, [g(a),g(b)], [f(a),f(b)], markersize=5, color=:orange)
plt
end
caption = L"""
Geometric interpretation of ``L=\lim_{x \rightarrow 0} x^2 / (\sqrt{1 +
x} - 1 - x^2)``. At ``0`` this limit is indeterminate of the form
``0/0``. The value for a fixed ``x`` can be seen as the slope of a secant
line of a parametric plot of the two functions, plotted as ``(g,
f)``. In this figure, the limiting "tangent" line has ``0`` slope,
corresponding to the limit ``L``. In general, L'Hospital's rule is
nothing more than a statement about slopes of tangent lines.
"""
n = 6
anim = @animate for i=1:n
lhopitals_picture_graph(i)
end
imgfile = tempname() * ".gif"
gif(anim, imgfile, fps = 1)
plotly()
ImageFile(imgfile, caption)
```
## Generalizations
L'Hospital's rule generalizes to other indeterminate forms, in
particular the indeterminate form ``\infty/\infty`` can be proved at the same time as ``0/0``
with a more careful
[proof](http://en.wikipedia.org/wiki/L%27H%C3%B4pital%27s_rule#General_proof).
The value ``c`` in the limit can also be infinite. Consider this case with ``c=\infty``:
```math
\begin{align*}
\lim_{x \rightarrow \infty} \frac{f(x)}{g(x)} &=
\lim_{x \rightarrow 0} \frac{f(1/x)}{g(1/x)}
\end{align*}
```
L'Hospital's limit applies as ``x \rightarrow 0``, so we differentiate to get:
```math
\begin{align*}
\lim_{x \rightarrow 0} \frac{[f(1/x)]'}{[g(1/x)]'}
&= \lim_{x \rightarrow 0} \frac{f'(1/x)\cdot(-1/x^2)}{g'(1/x)\cdot(-1/x^2)}\\
&= \lim_{x \rightarrow 0} \frac{f'(1/x)}{g'(1/x)}\\
&= \lim_{x \rightarrow \infty} \frac{f'(x)}{g'(x)},
\end{align*}
```
*assuming* the latter limit exists, L'Hospital's rule assures the equality
```math
\lim_{x \rightarrow \infty} \frac{f(x)}{g(x)} =
\lim_{x \rightarrow \infty} \frac{f'(x)}{g'(x)},
```
##### Examples
For example, consider
```math
\lim_{x \rightarrow \infty} \frac{x}{e^x}.
```
We see it is of the form ``\infty/\infty``. Taking advantage of the fact that L'Hospital's rule applies to limits
at ``\infty``, we have that this limit will exist and be equal to this one,
should it exist:
```math
\lim_{x \rightarrow \infty} \frac{1}{e^x}.
```
This limit is, of course, ``0``, as it is of the form ``1/\infty``. It is not
hard to build up from here to show that for any integer value of ``n>0``
that:
```math
\lim_{x \rightarrow \infty} \frac{x^n}{e^x} = 0.
```
This is an expression of the fact that exponential functions grow faster than polynomial functions.
Similarly, powers grow faster than logarithms, as this limit shows, which is indeterminate of the form ``\infty/\infty``:
```math
\lim_{x \rightarrow \infty} \frac{\log(x)}{x} =
\lim_{x \rightarrow \infty} \frac{1/x}{1} = 0,
```
the first equality by L'Hospital's rule, as the second limit exists.
## Other indeterminate forms
Indeterminate forms of the type ``0 \cdot \infty``, ``0^0``,
``\infty^\infty``, ``\infty - \infty`` can be re-expressed to be in the
form ``0/0`` or ``\infty/\infty`` and then L'Hospital's theorem can be
applied.
###### Example: rewriting ``0 \cdot \infty``
What is the limit ``x \log(x)`` as ``x \rightarrow 0+``? The form is ``0\cdot \infty``, rewriting, we see this is just:
```math
\lim_{x \rightarrow 0+}\frac{\log(x)}{1/x}.
```
L'Hospital's rule clearly applies to one-sided limits, as well as two
(our proof sketch used one-sided limits), so this limit will equal the
following, should it exist:
```math
\lim_{x \rightarrow 0+}\frac{1/x}{-1/x^2} = \lim_{x \rightarrow 0+} -x = 0.
```
###### Example: rewriting ``0^0``
What is the limit ``x^x`` as ``x \rightarrow 0+``? The expression is of the form ``0^0``, which is indeterminate. (Even though floating point math defines the value as ``1``.) We can rewrite this by taking a log:
```math
x^x = \exp(\log(x^x)) = \exp(x \log(x)) = \exp(\log(x)/(1/x)).
```
Be just saw that ``\lim_{x \rightarrow 0+}\log(x)/(1/x) = 0``. So by the
rules for limits of compositions and the fact that ``e^x`` is
continuous, we see ``\lim_{x \rightarrow 0+} x^x = e^0 = 1``.
##### Example: rewriting ``\infty - \infty``
A limit ``\lim_{x \rightarrow c} f(x) - g(x)`` of indeterminate form ``\infty - \infty`` can be reexpressed to be of the from ``0/0`` through the transformation:
```math
\begin{align*}
f(x) - g(x) &= f(x)g(x) \cdot (\frac{1}{g(x)} - \frac{1}{f(x)}) \\
&= \frac{\frac{1}{g(x)} - \frac{1}{f(x)}}{\frac{1}{f(x)g(x)}}.
\end{align*}
```
Applying this to
```math
L = \lim_{x \rightarrow 1} \big(\frac{x}{x-1} - \frac{1}{\log(x)}\big)
```
We get that ``L`` is equal to the following limit:
```math
\lim_{x \rightarrow 1} \frac{\log(x) - \frac{x-1}{x}}{\frac{x-1}{x} \log(x)}
=
\lim_{x \rightarrow 1} \frac{x\log(x)-(x-1)}{(x-1)\log(x)}
```
In `SymPy` we have:
```julia
𝒇 = x*log(x) - (x-1)
𝒈 = (x-1)*log(x)
𝒇(1), 𝒈(1)
```
L'Hospital's rule applies to the form ``0/0``, so we try:
```julia
𝒇′ = diff(𝒇, x)
𝒈′ = diff(𝒈, x)
𝒇′(1), 𝒈′(1)
```
Again, we get the indeterminate form ``0/0``, so we try again with second derivatives:
```julia
𝒇′′ = diff(𝒇, x, x)
𝒈′′ = diff(𝒈, x, x)
𝒇′′(1), 𝒈′′(1)
```
From this we see the limit is ``1/2``, as could have been done directly:
```julia
limit(𝒇/𝒈, x=>1)
```
## The assumptions are necessary
##### Example: the limit existing is necessary
The following limit is *easily* seen by comparing terms of largest growth:
```math
1 = \lim_{x \rightarrow \infty} \frac{x - \sin(x)}{x}
```
However, the limit of the ratio of the derivatives *does* not exist:
```math
\lim_{x \rightarrow \infty} \frac{1 - \cos(x)}{1},
```
as the function just oscillates. This shows that L'Hospital's rule does not apply when the limit of the the ratio of the derivatives does not exist.
##### Example: the assumptions matter
This example comes from the thesis of Gruntz to highlight possible issues when computer systems do simplifications.
Consider:
```math
\lim_{x \rightarrow \infty} \frac{1/2\sin(2x) +x}{\exp(\sin(x))\cdot(\cos(x)\sin(x)+x)}.
```
If we apply L'Hospital's rule using simplification we have:
```julia
u(x) = 1//2*sin(2x) + x
v(x) = exp(sin(x))*(cos(x)*sin(x) + x)
up, vp = diff(u(x),x), diff(v(x),x)
limit(simplify(up/vp), x => oo)
```
However, this answer is incorrect. The reason being subtle. The simplification cancels a term of ``\cos(x)`` that appears in the numerator and denominator. Before cancellation, we have `vp` will have infinitely many zero's as ``x`` approaches ``\infty`` so L'Hospital's won't apply (the limit won't exist, as every ``2\pi`` the ratio is undefined so the function is never eventually close to some ``L``).
This ratio has no limit, as it oscillates, as confirmed by `SymPy`:
```julia
limit(u(x)/v(x), x=> oo)
```
## Questions
###### Question
This function ``f(x) = \sin(5x)/x`` is *indeterminate* at ``x=0``. What type?
```julia; echo=false
lh_choices = [
"``0/0``",
"``\\infty/\\infty``",
"``0^0``",
"``\\infty - \\infty``",
"``0 \\cdot \\infty``"
]
nothing
```
```julia; hold=true; echo=false
ans = 1
radioq(lh_choices, ans, keep_order=true)
```
###### Question
This function ``f(x) = \sin(x)^{\sin(x)}`` is *indeterminate* at ``x=0``. What type?
```julia; hold=true; echo=false
ans =3
radioq(lh_choices, ans, keep_order=true)
```
###### Question
This function ``f(x) = (x-2)/(x^2 - 4)`` is *indeterminate* at ``x=2``. What type?
```julia; hold=true; echo=false
ans = 1
radioq(lh_choices, ans, keep_order=true)
```
###### Question
This function ``f(x) = (g(x+h) - g(x-h)) / (2h)`` (``g`` is continuous) is *indeterminate* at ``h=0``. What type?
```julia; hold=true; echo=false
ans = 1
radioq(lh_choices, ans, keep_order=true)
```
###### Question
This function ``f(x) = x \log(x)`` is *indeterminate* at ``x=0``. What type?
```julia; hold=true; echo=false
ans = 5
radioq(lh_choices, ans, keep_order=true)
```
###### Question
Does L'Hospital's rule apply to this limit:
```math
\lim_{x \rightarrow \pi} \frac{\sin(\pi x)}{\pi x}.
```
```julia; hold=true; echo=false
choices = [
"Yes. It is of the form ``0/0``",
"No. It is not indeterminate"
]
ans = 2
radioq(choices, ans)
```
###### Question
Use L'Hospital's rule to find the limit
```math
L = \lim_{x \rightarrow 0} \frac{4x - \sin(x)}{x}.
```
What is ``L``?
```julia; hold=true; echo=false
f(x) = (4x - sin(x))/x
L = float(N(limit(f, 0)))
numericq(L)
```
###### Question
Use L'Hospital's rule to find the limit
```math
L = \lim_{x \rightarrow 0} \frac{\sqrt{1+x} - 1}{x}.
```
What is ``L``?
```julia; hold=true; echo=false
f(x) = (sqrt(1+x) - 1)/x
L = float(N(limit(f, 0)))
numericq(L)
```
###### Question
Use L'Hospital's rule *one* or more times to find the limit
```math
L = \lim_{x \rightarrow 0} \frac{x - \sin(x)}{x^3}.
```
What is ``L``?
```julia; hold=true; echo=false
f(x) = (x - sin(x))/x^3
L = float(N(limit(f, 0)))
numericq(L)
```
###### Question
Use L'Hospital's rule *one* or more times to find the limit
```math
L = \lim_{x \rightarrow 0} \frac{1 - x^2/2 - \cos(x)}{x^3}.
```
What is ``L``?
```julia; hold=true; echo=false
f(x) = (1 - x^2/2 - cos(x))/x^3
L = float(N(limit(f, 0)))
numericq(L)
```
###### Question
Use L'Hospital's rule *one* or more times to find the limit
```math
L = \lim_{x \rightarrow \infty} \frac{\log(\log(x))}{\log(x)}.
```
What is ``L``?
```julia; hold=true; echo=false
f(x) = log(log(x))/log(x)
L = N(limit(f(x), x=> oo))
numericq(L)
```
###### Question
By using a common denominator to rewrite this expression, use L'Hospital's rule to find the limit
```math
L = \lim_{x \rightarrow 0} \frac{1}{x} - \frac{1}{\sin(x)}.
```
What is ``L``?
```julia; hold=true; echo=false
f(x) = 1/x - 1/sin(x)
L = float(N(limit(f, 0)))
numericq(L)
```
##### Question
Use L'Hospital's rule to find the limit
```math
L = \lim_{x \rightarrow \infty} \log(x)/x
```
What is ``L``?
```julia; hold=true; echo=false
L = float(N(limit(log(x)/x, x=>oo)))
numericq(L)
```
##### Question
Using L'Hospital's rule, does
```math
\lim_{x \rightarrow 0+} x^{\log(x)}
```
exist?
Consider ``x^{\log(x)} = e^{\log(x)\log(x)}``.
```julia; hold=true; echo=false
yesnoq(false)
```
##### Question
Using L'Hospital's rule, find the limit of
```math
\lim_{x \rightarrow 1} (2-x)^{\tan(\pi/2 \cdot x)}.
```
(Hint, express as ``\exp^{\tan(\pi/2 \cdot x) \cdot \log(2-x)}`` and take the limit of the resulting exponent.)
```julia; hold=true; echo=false
choices = [
"``e^{2/\\pi}``",
"``{2\\pi}``",
"``1``",
"``0``",
"It does not exist"
]
ans =1
radioq(choices, ans)
```

806
CwJ/derivatives/linearization.jmd Normal file
View File

@@ -0,0 +1,806 @@
# Linearization
This section uses these add-on packages:
```julia
using CalculusWithJulia
using Plots
using SymPy
using TaylorSeries
using DualNumbers
```
```julia; echo=false; results="hidden"
using CalculusWithJulia.WeaveSupport
const frontmatter = (
title = "Linearization",
description = "Calculus with Julia: Linearization",
tags = ["CalculusWithJulia", "derivatives", "linearization"],
);
nothing
```
----
The derivative of $f(x)$ has the interpretation as the slope of the
tangent line. The tangent line is the line that best approximates the
function at the point.
Using the point-slope form of a line, we see that the tangent line to the graph of $f(x)$ at $(c,f(c))$ is given by:
```math
y = f(c) + f'(c) \cdot (x - c).
```
This is written as an equation, though we prefer to work with
functions within `Julia`. Here we write such a function as an
operator - it takes a function `f` and returns a function
representing the tangent line.
```julia; eval=false
tangent(f, c) = x -> f(c) + f'(c) * (x - c)
```
(Recall, the `->` indicates that an anonymous function is being generated.)
This function along with the `f'` notation for automatic derivatives is defined in the
`CalculusWithJulia` package.
We make some graphs with tangent lines:
```julia;hold=true
f(x) = x^2
plot(f, -3, 3)
plot!(tangent(f, -1))
plot!(tangent(f, 2))
```
The graph shows that near the point, the line and function are close,
but this need not be the case away from the point. We can express this informally as
```math
f(x) \approx f(c) + f'(c) \cdot (x-c)
```
with the understanding this applies for $x$ "close" to $c$.
Usually for the applications herein, instead of ``x`` and ``c`` the two points are ``x+\Delta_x`` and ``x``. This gives:
> *Linearization*: ``\Delta_y = f(x +\Delta_x) - f(x) \approx f'(x) \Delta_x``, for small ``\Delta_x``.
This section gives some implications of this fact and quantifies what
"close" can mean.
##### Example
There are several approximations that are well known in physics, due to their widespread usage:
* That $\sin(x) \approx x$ around $x=0$:
```julia;hold=true
plot(sin, -pi/2, pi/2)
plot!(tangent(sin, 0))
```
Symbolically:
```julia; hold=true
@syms x
c = 0
f(x) = sin(x)
f(c) + diff(f(x),x)(c) * (x - c)
```
* That $\log(1 + x) \approx x$ around $x=0$:
```julia; hold=true;
f(x) = log(1 + x)
plot(f, -1/2, 1/2)
plot!(tangent(f, 0))
```
Symbolically:
```julia; hold=true
@syms x
c = 0
f(x) = log(1 + x)
f(c) + diff(f(x),x)(c) * (x - c)
```
(The `log1p` function implements a more accurate version of this function when numeric values are needed.)
* That $1/(1-x) \approx x$ around $x=0$:
```julia; hold=true;
f(x) = 1/(1-x)
plot(f, -1/2, 1/2)
plot!(tangent(f, 0))
```
Symbolically:
```julia; hold=true
@syms x
c = 0
f(x) = 1 / (1 - x)
f(c) + diff(f(x),x)(c) * (x - c)
```
* That ``(1+x)^n \approx 1 + nx`` around ``x = 0``. For example, with ``n=5``
```julia; hold=true;
n = 5
f(x) = (1+x)^n # f'(0) = n = n(1+x)^(n-1) at x=0
plot(f, -1/2, 1/2)
plot!(tangent(f, 0))
```
Symbolically:
```julia; hold=true
@syms x, n::real
c = 0
f(x) = (1 + x)^n
f(c) + diff(f(x),x)(x=>c) * (x - c)
```
----
In each of these cases, a more complicated non-linear function
is well approximated in a region of interest by a simple linear
function.
## Numeric approximations
```julia; hold=true; echo=false
f(x) = sin(x)
a, b = -1/4, pi/2
p = plot(f, a, b, legend=false);
plot!(p, x->x, a, b);
plot!(p, [0,1,1], [0, 0, 1], color=:brown);
plot!(p, [1,1], [0, sin(1)], color=:green, linewidth=4);
annotate!(p, collect(zip([1/2, 1+.075, 1/2-1/8], [.05, sin(1)/2, .75], ["Δx", "Δy", "m=dy/dx"])));
p
```
The plot shows the tangent line with slope $dy/dx$ and the actual
change in $y$, $\Delta y$, for some specified $\Delta x$. The small
gap above the sine curve is the error were the value of the sine approximated using the drawn tangent line. We can see that approximating
the value of $\Delta y = \sin(c+\Delta x) - \sin(c)$ with the often
easier to compute $(dy/dx) \cdot \Delta x = f'(c)\Delta x$ - for small enough values of
$\Delta x$ - is not going to be too far off provided $\Delta x$ is not too large.
This approximation is known as linearization. It can be used both in
theoretical computations and in pratical applications. To see how
effective it is, we look at some examples.
##### Example
If $f(x) = \sin(x)$, $c=0$ and $\Delta x= 0.1$ then the values for the actual change in the function values and the value of $\Delta y$ are:
```julia;
f(x) = sin(x)
c, deltax = 0, 0.1
f(c + deltax) - f(c), f'(c) * deltax
```
The values are pretty close. But what is $0.1$ radians? Lets use degrees. Suppose we have $\Delta x = 10^\circ$:
```julia;
deltax⁰ = 10*pi/180
actual = f(c + deltax⁰) - f(c)
approx = f'(c) * deltax⁰
actual, approx
```
They agree until the third decimal value. The *percentage error* is just $1/2$ a percent:
```julia;
(approx - actual) / actual * 100
```
### Relative error or relative change
The relative error is defined by
```math
\big| \frac{\text{actual} - \text{approximate}}{\text{actual}} \big|.
```
However, typically with linearization, we talk about the *relative change*, not relative error, as the denominator is easier to compute. This is
```math
\frac{f(x + \Delta_x) - f(x)}{f(x)} = \frac{\Delta_y}{f(x)} \approx
\frac{f'(x) \cdot \Delta_x}{f(x)}
```
The *percentage change* multiplies by ``100``.
##### Example
What is the relative change in surface area of a sphere if the radius changes from ``r`` to ``r + dr``?
We have ``S = 4\pi r^2`` so the approximate relative change, ``dy/S`` is given, using the derivative ``dS/dr = 8\pi r``, by
```math
\frac{8\pi\cdot r\cdot dr}{4\pi r^2} = 2r\cdot dr.
```
##### Example
We are traveling ``60`` miles. At ``60`` miles an hour, we will take ``60`` minutes (or one hour). How long will it take at ``70`` miles an hour? (Assume you can't divide, but, instead, can only multiply!)
Well the answer is $60/70$ hours or $60/70 \cdot 60$ minutes. But we
can't divide, so we turn this into a multiplication problem via some algebra:
```math
\frac{60}{70} = \frac{60}{60 + 10} = \frac{1}{1 + 10/60} = \frac{1}{1 + 1/6}.
```
Okay, so far no calculator was needed. We wrote $70 = 60 + 10$, as we
know that $60/60$ is just $1$. This almost gets us there. If we really
don't want to divide, we can get an answer by using the tangent line
approximation for $1/(1+x)$ around $x=0$. This is $1/(1+x) \approx 1 -
x$. (You can check by finding that $f'(0) = -1$.) Thus, our answer is
approximately $5/6$ of an hour or 50 minutes.
How much in error are we?
```julia;
abs(50 - 60/70*60) / (60/70*60) * 100
```
That's about $3$ percent. Not bad considering we could have done all
the above in our head while driving without taking our eyes off the
road to use the calculator on our phone for a division.
##### Example
A ``10``cm by ``10``cm by ``10``cm cube will contain ``1`` liter
(``1000``cm``^3``). In manufacturing such a cube, the side lengths are
actually $10.1$ cm. What will be the volume in liters? Compute this
with a linear approximation to $(10.1)^3$.
Here $f(x) = x^3$ and we are asked to approximate $f(10.1)$. Letting $c=10$, we have:
```math
f(c + \Delta) \approx f(c) + f'(c) \cdot \Delta = 1000 + f'(c) \cdot (0.1)
```
Computing the derivative can be done easily, we get for our answer:
```julia;
fp(x) = 3*x^2
c₀, Delta = 10, 0.1
approx₀ = 1000 + fp(c₀) * Delta
```
This is a relative error as a percent of:
```julia;
actual₀ = 10.1^3
(actual₀ - approx₀)/actual₀ * 100
```
The manufacturer may be interested instead in comparing the volume of the actual object to the $1$ liter target. They might use the approximate value for this comparison, which would yield:
```julia;
(1000 - approx₀)/approx₀ * 100
```
This is off by about $3$ percent. Not so bad for some applications, devastating for others.
##### Example: Eratosthenes and the circumference of the earth
[Eratosthenes](https://en.wikipedia.org/wiki/Eratosthenes) is said to have been the first person to estimate the radius (or by relation the circumference) of the earth. The basic idea is based on the difference of shadows cast by the sun. Suppose Eratosthenes sized the circumference as ``252,000`` *stadia*. Taking ``1``` stadia as ``160`` meters and the actual radius of the earth as ``6378.137`` kilometers, we can convert to see that Eratosthenes estimated the radius as ``6417``.
If Eratosthenes were to have estimated the volume of a spherical earth, what would be his approximate percentage change between his estimate and the actual?
Using ``V = 4/3 \pi r^3`` we get ``V' = 4\pi r^2``:
```julia
rₑ = 6417
rₐ = 6378.137
Δᵣ = rₑ - rₐ
Vₛ(r) = 4/3 * pi * r^3
Δᵥ = Vₛ'(rₑ) * Δᵣ
Δᵥ / Vₛ(rₑ) * 100
```
##### Example: a simple pendulum
A *simple* pendulum is comprised of a massless "bob" on a rigid "rod"
of length $l$. The rod swings back and forth making an angle $\theta$
with the perpendicular. At rest $\theta=0$, here we have $\theta$ swinging with $\lvert\theta\rvert \leq \theta_0$
for some $\theta_0$.
According to [Wikipedia](http://tinyurl.com/yz5sz7e) - and many
introductory physics book - while swinging, the angle $\theta$ varies
with time following this equation:
```math
\theta''(t) + \frac{g}{l} \sin(\theta(t)) = 0.
```
That is, the second derivative of $\theta$ is proportional to the sine
of $\theta$ where the proportionality constant involves $g$ from
gravity and the length of the "rod."
This would be much easier if the second derivative were proportional to the angle $\theta$ and not its sine.
[Huygens](http://en.wikipedia.org/wiki/Christiaan_Huygens) used the
approximation of $\sin(x) \approx x$, noted above, to say that when
the angle is not too big, we have the pendulum's swing obeying
$\theta''(t) = -g/l \cdot t$. Without getting too involved in why,
we can verify by taking two derivatives that $\theta_0\sin(\sqrt{g/l}\cdot t)$ will be a solution to this
modified equation.
With this solution, the motion is periodic with constant amplitude (assuming frictionless behaviour), as
the sine function is. More surprisingly, the period is found from $T =
2\pi/(\sqrt{g/l}) = 2\pi \sqrt{l/g}$. It depends on $l$ - longer
"rods" take more time to swing back and forth - but does not depend
on the how wide the pendulum is swinging between (provided $\theta_0$
is not so big the approximation of $\sin(x) \approx x$ fails). This
latter fact may be surprising, though not to Galileo who discovered
it.
## Differentials
The Leibniz notation for a derivative is ``dy/dx`` indicating the
change in ``y`` as ``x`` changes. It proves convenient to decouple
this using *differentials* ``dx`` and ``dy``. What do these notations
mean? They measure change along the tangent line in same way
``\Delta_x`` and ``\Delta_y`` measure change for the function. The differential ``dy`` depends on both ``x`` and ``dx``, it being defined by ``dy=f'(x)dx``. As tangent lines locally represent a function, ``dy`` and ``dx`` are often associated with an *infinitesimal* difference.
Taking ``dx = \Delta_x``, as in the previous graphic, we can compare ``dy`` -- the change along the tangent line given by ``dy/dx \cdot dx`` -- and ``\Delta_y`` -- the change along the function given by ``f(x + \Delta_x) - f(x)``. The linear approximation, ``f(x + \Delta_x) - f(x)\approx f'(x)dx``, says that
```math
\Delta_y \approx dy; \quad \text{ when } \Delta_x = dx
```
## The error in approximation
How good is the approximation? Graphically we can see it is pretty
good for the graphs we choose, but are there graphs out there for
which the approximation is not so good? Of course. However, we can
say this (the
[Lagrange](http://en.wikipedia.org/wiki/Taylor%27s_theorem) form of a
more general Taylor remainder theorem):
> Let ``f(x)`` be twice differentiable on ``I=(a,b)``,
>``f`` is continuous on ``[a,b]``, and
> ``a < c < b``. Then for any ``x`` in ``I``, there exists some value ``\xi`` between ``c`` and ``x`` such that
> ``f(x) = f(c) + f'(c)(x-c) + (f''(\xi)/2)\cdot(x-c)^2``.
That is, the error is basically a constant depending on the concavity
of $f$ times a quadratic function centered at $c$.
For $\sin(x)$ at $c=0$ we get $\lvert\sin(x) - x\rvert = \lvert-\sin(\xi)\cdot x^2/2\rvert$.
Since $\lvert\sin(\xi)\rvert \leq 1$, we must have this bound:
$\lvert\sin(x) - x\rvert \leq x^2/2$.
Can we verify? Let's do so graphically:
```julia; hold=true
h(x) = abs(sin(x) - x)
g(x) = x^2/2
plot(h, -2, 2, label="h")
plot!(g, -2, 2, label="f")
```
The graph shows a tight bound near ``0`` and then a bound over this viewing window.
Similarly, for $f(x) = \log(1 + x)$ we have the following at $c=0$:
```math
f'(x) = 1/(1+x), \quad f''(x) = -1/(1+x)^2.
```
So, as $f(c)=0$ and $f'(c) = 1$, we have
```math
\lvert f(x) - x\rvert \leq \lvert f''(\xi)\rvert \cdot \frac{x^2}{2}
```
We see that $\lvert f''(x)\rvert$ is decreasing for $x > -1$. So if $-1 < x < c$ we have
```math
\lvert f(x) - x\rvert \leq \lvert f''(x)\rvert \cdot \frac{x^2}{2} = \frac{x^2}{2(1+x)^2}.
```
And for $c=0 < x$, we have
```math
\lvert f(x) - x\rvert \leq \lvert f''(0)\rvert \cdot \frac{x^2}{2} = x^2/2.
```
Plotting we verify the bound on ``|\log(1+x)-x|``:
```julia; hold=true
h(x) = abs(log(1+x) - x)
g(x) = x < 0 ? x^2/(2*(1+x)^2) : x^2/2
plot(h, -0.5, 2, label="h")
plot!(g, -0.5, 2, label="g")
```
Again, we see the very close bound near ``0``, which widens at the edges of the viewing window.
### Why is the remainder term as it is?
To see formally why the remainder is as it is, we recall the mean value
theorem in the extended form of Cauchy. Suppose $c=0$, $x > 0$, and let $h(x) = f(x) - (f(0) +
f'(0) x)$ and $g(x) = x^2$. Then we have that there exists a $e$ with
$0 < e < x$ such that
```math
\text{error} = h(x) - h(0) = (g(x) - g(0)) \frac{h'(e)}{g'(e)} = x^2 \cdot \frac{1}{2} \cdot \frac{f'(e) - f'(0)}{e} =
x^2 \cdot \frac{1}{2} \cdot f''(\xi).
```
The value of $\xi$, from the mean value theorem applied to $f'(x)$,
satisfies $0 < \xi < e < x$, so is in $[0,x].$
### The big (and small) "oh"
`SymPy` can find the tangent line expression as a special case of its `series` function (which implements [Taylor series](../taylor_series_polynomials.html)). The `series` function needs an expression to approximate; a variable specified, as there may be parameters in the expression; a value ``c`` for *where* the expansion is taken, with default ``0``; and a number of terms, for this example ``2`` for a constant and linear term. (There is also an optional `dir` argument for one-sided expansions.)
Here we see the answer provided for $e^{\sin(x)}$:
```julia;
@syms x
series(exp(sin(x)), x, 0, 2)
```
The expression $1 + x$ comes from the fact that `exp(sin(0))` is $1$, and the derivative `exp(sin(0)) * cos(0)` is *also* $1$. But what is the $\mathcal{O}(x^2)$?
We know the answer is *precisely* $f''(\xi)/2 \cdot x^2$ for some ``\xi``, but were we
only concerned about the scale as $x$ goes to zero
that when ``f''`` is continuous that the error when divided by ``x^2`` goes to some finite value (``f''(0)/2``). More generally, if the error divided by ``x^2`` is *bounded* as ``x`` goes to ``0``, then we say the error is "big oh" of ``x^2``.
The [big](http://en.wikipedia.org/wiki/Big_O_notation) "oh" notation,
``f(x) = \mathcal{O}(g(x))``, says that the ratio ``f(x)/g(x)`` is
bounded as ``x`` goes to ``0`` (or some other value ``c``, depending
on the context). A little "oh" (e.g., ``f(x) = \mathcal{o}(g(x))``)
would mean that the limit ``f(x)/g(x)`` would be ``0``, as
``x\rightarrow 0``, a much stronger assertion.
Big "oh" and little "oh" give us a sense of how good an approximation
is without being bogged down in the details of the exact value. As
such they are useful guides in focusing on what is primary and what is
secondary. Applying this to our case, we have this rough form of the
tangent line approximation valid for functions having a continuous second
derivative at ``c``:
```math
f(x) = f(c) + f'(c)(x-c) + \mathcal{O}((x-c)^2).
```
##### Example: the algebra of tangent line approximations
Suppose $f(x)$ and $g(x)$ are represented by their tangent lines about $c$, respectively:
```math
\begin{align*}
f(x) &= f(c) + f'(c)(x-c) + \mathcal{O}((x-c)^2), \\
g(x) &= g(c) + g'(c)(x-c) + \mathcal{O}((x-c)^2).
\end{align*}
```
Consider the sum, after rearranging we have:
```math
\begin{align*}
f(x) + g(x) &= \left(f(c) + f'(c)(x-c) + \mathcal{O}((x-c)^2)\right) + \left(g(c) + g'(c)(x-c) + \mathcal{O}((x-c)^2)\right)\\
&= \left(f(c) + g(c)\right) + \left(f'(c)+g'(c)\right)(x-c) + \mathcal{O}((x-c)^2).
\end{align*}
```
The two big "Oh" terms become just one as the sum of a constant times $(x-c)^2$ plus a constant time $(x-c)^2$ is just some other constant times $(x-c)^2$. What we can read off from this is the term multiplying $(x-c)$ is just the derivative of $f(x) + g(x)$ (from the sum rule), so this too is a tangent line approximation.
Is it a coincidence that a basic algebraic operation with tangent lines approximations produces a tangent line approximation? Let's try multiplication:
```math
\begin{align*}
f(x) \cdot g(x) &= [f(c) + f'(c)(x-c) + \mathcal{O}((x-c)^2)] \cdot [g(c) + g'(c)(x-c) + \mathcal{O}((x-c)^2)]\\
&=[(f(c) + f'(c)(x-c)] \cdot [g(c) + g'(c)(x-c)] + (f(c) + f'(c)(x-c) \cdot \mathcal{O}((x-c)^2)) + g(c) + g'(c)(x-c) \cdot \mathcal{O}((x-c)^2)) + [\mathcal{O}((x-c)^2))]^2\\
&= [(f(c) + f'(c)(x-c)] \cdot [g(c) + g'(c)(x-c)] + \mathcal{O}((x-c)^2)\\
&= f(c) \cdot g(c) + [f'(c)\cdot g(c) + f(c)\cdot g'(c)] \cdot (x-c) + [f'(c)\cdot g'(c) \cdot (x-c)^2 + \mathcal{O}((x-c)^2)] \\
&= f(c) \cdot g(c) + [f'(c)\cdot g(c) + f(c)\cdot g'(c)] \cdot (x-c) + \mathcal{O}((x-c)^2)
\end{align*}
```
The big "oh" notation just sweeps up many things including any products of it *and* the term $f'(c)\cdot g'(c) \cdot (x-c)^2$. Again, we see from the product rule that this is just a tangent line approximation for $f(x) \cdot g(x)$.
The basic mathematical operations involving tangent lines can be computed just using the tangent lines when the desired accuracy is at the tangent line level. This is even true for composition, though there the outer and inner functions may have different "$c$"s.
Knowing this can simplify the task of finding tangent line approximations of compound expressions.
For example, suppose we know that at $c=0$ we have these formula where $a \approx b$ is a shorthand for the more formal $a=b + \mathcal{O}(x^2)$:
```math
\sin(x) \approx x, \quad e^x \approx 1 + x, \quad \text{and}\quad 1/(1+x) \approx 1 - x.
```
Then we can immediately see these tangent line approximations about $x=0$:
```math
e^x \cdot \sin(x) \approx (1+x) \cdot x = x + x^2 \approx x,
```
and
```math
\frac{\sin(x)}{e^x} \approx \frac{x}{1 + x} \approx x \cdot(1-x) = x-x^2 \approx x.
```
Since $\sin(0) = 0$, we can use these to find the tangent line approximation of
```math
e^{\sin(x)} \approx e^x \approx 1 + x.
```
Note that $\sin(\exp(x))$ is approximately $\sin(1+x)$ but not approximately $1+x$, as the expansion for $\sin$ about $1$ is not simply $x$.
### The TaylorSeries package
The `TaylorSeries` packages will do these calculations in a manner similar to how `SymPy` transforms a function and a symbolic variable into a symbolic expression.
For example, we have
```julia
t = Taylor1(Float64, 1)
```
The number type and the order is specified to the constructor. Linearization is order ``1``, other orders will be discussed later. This variable can now be composed with mathematical functions and the linearization of the function will be returned:
```julia
sin(t), exp(t), 1/(1+t)
```
```julia
sin(t)/exp(t), exp(sin(t))
```
##### Example: Automatic differentiation
Automatic differentiation (forward mode) essentially uses this technique. A "dual" is introduced which has terms ``a +b\epsilon`` where ``\epsilon^2 = 0``.
The ``\epsilon`` is like ``x`` in a linear expansion, so the `a` coefficient encodes the value and the `b` coefficient reflects the derivative at the value. Numbers are treated like a variable, so their "b coefficient" is a `1`. Here then is how `0` is encoded:
```julia;
Dual(0, 1)
```
Then what is ``\(x)``? It should reflect both ``(\sin(0), \cos(0))`` the latter being the derivative of ``\sin``. We can see this is *almost* what is computed behind the scenes through:
```julia; hold=true
x = Dual(0, 1)
@code_lowered sin(x)
```
This output of `@code_lowered` can be confusing, but this simple case needn't be. Working from the end we see an assignment to a variable named `%7` of `Dual(%3, %6)`. The value of `%3` is `sin(x)` where `x` is the value `0` above. The value of `%6` is `cos(x)` *times* the value `1` above (the `xp`), which reflects the *chain* rule being used. (The derivative of `sin(u)` is `cos(u)*du`.) So this dual number encodes both the function value at `0` and the derivative of the function at `0`.)
Similarly, we can see what happens to `log(x)` at `1` (encoded by `Dual(1,1)`):
```julia; hold=true
x = Dual(1, 1)
@code_lowered log(x)
```
We can see the derivative again reflects the chain rule, it being given by `1/x * xp` where `xp` acts like `dx` (from assignments `%5` and `%4`). Comparing the two outputs, we see only the assignment to `%4` differs, it reflecting the derivative of the function.
## Questions
###### Question
What is the right linear approximation for $\sqrt{1 + x}$ near $0$?
```julia; hold=true; echo=false
choices = [
"``1 + 1/2``",
"``1 + x^{1/2}``",
"``1 + (1/2) \\cdot x``",
"``1 - (1/2) \\cdot x``"]
ans = 3
radioq(choices, ans)
```
###### Question
What is the right linear approximation for $(1 + x)^k$ near $0$?
```julia; hold=true; echo=false
choices = [
"``1 + k``",
"``1 + x^k``",
"``1 + k \\cdot x``",
"``1 - k \\cdot x``"]
ans = 3
radioq(choices, ans)
```
###### Question
What is the right linear approximation for $\cos(\sin(x))$ near $0$?
```julia; hold=true; echo=false
choices = [
"``1``",
"``1 + x``",
"``x``",
"``1 - x^2/2``"
]
ans = 1
radioq(choices, ans)
```
###### Question
What is the right linear approximation for $\tan(x)$ near $0$?
```julia; hold=true; echo=false
choices = [
"``1``",
"``x``",
"``1 + x``",
"``1 - x``"
]
ans = 2
radioq(choices, ans)
```
###### Question
What is the right linear approximation of $\sqrt{25 + x}$ near $x=0$?
```julia; hold=true; echo=false
choices = [
"``5 \\cdot (1 + (1/2) \\cdot (x/25))``",
"``1 - (1/2) \\cdot x``",
"``1 + x``",
"``25``"
]
ans = 1
radioq(choices, ans)
```
###### Question
Let $f(x) = \sqrt{x}$. Find the actual error in approximating $f(26)$ by the
value of the tangent line at $(25, f(25))$ at $x=26$.
```julia; hold=true; echo=false
tgent(x) = 5 + x/10
ans = tgent(1) - sqrt(26)
numericq(ans)
```
###### Question
An estimate of some quantity was $12.34$ the actual value was $12$. What was the *percentage error*?
```julia; hold=true; echo=false
est = 12.34
act = 12.0
ans = (est -act)/act * 100
numericq(ans)
```
###### Question
Find the percentage error in estimating $\sin(5^\circ)$ by $5 \pi/180$.
```julia; hold=true; echo=false
tl(x) = x
x0 = 5 * pi/180
est = x0
act = sin(x0)
ans = (est -act)/act * 100
numericq(ans)
```
###### Question
The side length of a square is measured roughly to be $2.0$ cm. The actual length $2.2$ cm. What is the difference in area (in absolute values) as *estimated* by a tangent line approximation.
```julia; hold=true; echo=false
tl(x) = 4 + 4x
ans = tl(.2) - 4
numericq(abs(ans))
```
###### Question
The [Birthday problem](https://en.wikipedia.org/wiki/Birthday_problem) computes the probability that in a group of ``n`` people, under some assumptions, that no two share a birthday. Without trying to spoil the problem, we focus on the calculus specific part of the problem below:
```math
\begin{align*}
p
&= \frac{365 \cdot 364 \cdot \cdots (365-n+1)}{365^n} \\
&= \frac{365(1 - 0/365) \cdot 365(1 - 1/365) \cdot 365(1-2/365) \cdot \cdots \cdot 365(1-(n-1)/365)}{365^n}\\
&= (1 - \frac{0}{365})\cdot(1 -\frac{1}{365})\cdot \cdots \cdot (1-\frac{n-1}{365}).
\end{align*}
```
Taking logarithms, we have ``\log(p)`` is
```math
\log(1 - \frac{0}{365}) + \log(1 -\frac{1}{365})+ \cdots + \log(1-\frac{n-1}{365}).
```
Now, use the tangent line approximation for ``\log(1 - x)`` and the sum formula for ``0 + 1 + 2 + \dots + (n-1)`` to simplify the value of ``\log(p)``:
```julia; hold=true; echo=false
choices = ["``-n(n-1)/2/365``",
"``-n(n-1)/2\\cdot 365``",
"``-n^2/(2\\cdot 365)``",
"``-n^2 / 2 \\cdot 365``"]
radioq(choices, 1, keep_order=true)
```
If ``n = 10``, what is the approximation for ``p`` (not ``\log(p)``)?
```julia; hold=true; echo=false
n=10
val = exp(-n*(n-1)/2/365)
numericq(val)
```
If ``n=100``, what is the approximation for ``p`` (not ``\log(p)``?
```julia; hold=true; echo=false
n=100
val = exp(-n*(n-1)/2/365)
numericq(val, 1e-2)
```

23
CwJ/derivatives/mean-value.js Normal file
View File

@@ -0,0 +1,23 @@
// https://jsxgraph.uni-bayreuth.de/wiki/index.php?title=Mean_Value_Theorem
var board = JXG.JSXGraph.initBoard('jsxgraph', {boundingbox: [-5, 10, 7, -6], axis:true});
board.suspendUpdate();
var p = [];
p[0] = board.create('point', [-1,-2], {size:2});
p[1] = board.create('point', [6,5], {size:2});
p[2] = board.create('point', [-0.5,1], {size:2});
p[3] = board.create('point', [3,3], {size:2});
var f = JXG.Math.Numerics.lagrangePolynomial(p);
var graph = board.create('functiongraph', [f,-10, 10]);
var g = function(x) {
return JXG.Math.Numerics.D(f)(x)-(p[1].Y()-p[0].Y())/(p[1].X()-p[0].X());
};
var r = board.create('glider', [
function() { return JXG.Math.Numerics.root(g,(p[0].X()+p[1].X())*0.5); },
function() { return f(JXG.Math.Numerics.root(g,(p[0].X()+p[1].X())*0.5)); },
graph], {name:' ',size:4,fixed:true});
board.create('tangent', [r], {strokeColor:'#ff0000'});
line = board.create('line',[p[0],p[1]],{strokeColor:'#ff0000',dash:1});
board.unsuspendUpdate();

693
CwJ/derivatives/mean_value_theorem.jmd Normal file
View File

@@ -0,0 +1,693 @@
# The mean value theorem for differentiable functions.
This section uses these add-on packages:
```julia
using CalculusWithJulia
using Plots
using Roots
```
```julia; echo=false; results="hidden"
using CalculusWithJulia.WeaveSupport
using Printf
using SymPy
fig_size = (600, 400)
const frontmatter = (
title = "The mean value theorem for differentiable functions.",
description = "Calculus with Julia: The mean value theorem for differentiable functions.",
tags = ["CalculusWithJulia", "derivatives", "the mean value theorem for differentiable functions."],
);
nothing
```
----
A function is *continuous* at $c$ if $f(c+h) - f(c) \rightarrow 0$ as $h$ goes to $0$. We can write that as ``f(c+h) - f(x) = \epsilon_h``, with ``\epsilon_h`` denoting a function going to ``0`` as ``h \rightarrow 0``. With this notion, differentiability could be written as ``f(c+h) - f(c) - f'(c)h = \epsilon_h \cdot h``. This is clearly a more demanding requirement that mere continuity at ``c``.
We defined a function to be *continuous* on an interval $I=(a,b)$ if
it was continuous at each point $c$ in $I$. Similarly, we define a
function to be *differentiable* on the interval $I$ it it is differentiable
at each point $c$ in $I$.
This section looks at properties of differentiable functions. As there is a more stringent definition, perhaps more properties are a consequence of the definition.
## Differentiable is more restrictive than continuous.
Let $f$ be a differentiable function on $I=(a,b)$. We see that
``f(c+h) - f(c) = f'(c)h + \epsilon_h\cdot h = h(f'(c) + \epsilon_h)``. The right hand side will clearly go to ``0`` as ``h\rightarrow 0``, so ``f`` will be continuous. In short:
> A differentiable function on $I=(a,b)$ is continuous on $I$.
Is it possible that all continuous functions are differentiable?
The fact that the derivative is related to the tangent line's slope
might give an indication that this won't be the case - we just need a
function which is continuous but has a point with no tangent line. The
usual suspect is $f(x) = \lvert x\rvert$ at $0$.
```julia; hold=true
f(x) = abs(x)
plot(f, -1,1)
```
We can see formally that the secant line expression will not have a
limit when $c=0$ (the left limit is $-1$, the right limit $1$). But
more insight is gained by looking a the shape of the graph. At the origin, the graph
always is vee-shaped. There is no linear function that approximates this function
well. The function is just not smooth enough, as it has a kink.
There are other functions that have kinks. These are often associated
with powers. For example, at $x=0$ this function will not have a
derivative:
```julia; hold=true;
f(x) = (x^2)^(1/3)
plot(f, -1, 1)
```
Other functions have tangent lines that become vertical. The natural slope would be $\infty$, but this isn't a limiting answer (except in the extended sense we don't apply to the definition of derivatives). A candidate for this case is the cube root function:
```julia;
plot(cbrt, -1, 1)
```
The derivative at $0$ would need to be $+\infty$ to match the
graph. This is implied by the formula for the derivative from the
power rule: $f'(x) = 1/3 \cdot x^{-2/3}$, which has a vertical
asymptote at $x=0$.
```julia; echo=false
note("""
The `cbrt` function is used above, instead of `f(x) = x^(1/3)`, as the
latter is not defined for negative `x`. Though it can be for the exact
power `1/3`, it can't be for an exact power like `1/2`. This means the
value of the argument is important in determining the type of the
output - and not just the type of the argument. Having type-stable
functions is part of the magic to making `Julia` run fast, so `x^c` is
not defined for negative `x` and most floating point exponents.
""")
```
Lest you think that continuous functions always have derivatives
except perhaps at exceptional points, this isn't the case. The
functions used to
[model](http://tinyurl.com/cpdpheb) the
stock market are continuous but have no points where they are
differentiable.
## Derivatives and maxima.
We have defined an *absolute maximum* of $f(x)$ over an interval to be
a value $f(c)$ for a point $c$ in the interval that is as large as any
other value in the interval. Just specifying a function and an
interval does not guarantee an absolute maximum, but specifying a
*continuous* function and a *closed* interval does, by the extreme value theorem.
> *A relative maximum*: We say $f(x)$ has a *relative maximum* at $c$
> if there exists *some* interval $I=(a,b)$ with $a < c < b$ for which
> $f(c)$ is an absolute maximum for $f$ and $I$.
The difference is a bit subtle, for an absolute maximum the interval
must also be specified, for a relative maximum there just needs to
exist some interval, possibly really small, though it must be bigger
than a point.
```julia; echo=false
note("""
A hiker can appreciate the difference. A relative maximum would be the
crest of any hill, but an absolute maximum would be the summit.
""")
```
What does this have to do with derivatives?
[Fermat](http://science.larouchepac.com/fermat/fermat-maxmin.pdf),
perhaps with insight from Kepler, was interested in maxima of
polynomial functions. As a warm up, he considered a line segment $AC$ and a point $E$
with the task of choosing $E$ so that $(E-A) \times (C-A)$ being a maximum. We might recognize this as
finding the maximum of $f(x) = (x-A)\cdot(C-x)$ for some $A <
C$. Geometrically, we know this to be at the midpoint, as the equation
is a parabola, but Fermat was interested in an algebraic solution that
led to more generality.
He takes $b=AC$ and $a=AE$. Then the product is $a \cdot (b-a) =
ab - a^2$. He then perturbs this writing $AE=a+e$, then this new
product is $(a+e) \cdot (b - a - e)$. Equating the two, and canceling
like terms gives $be = 2ae + e^2$. He cancels the $e$ and basically
comments that this must be true for all $e$ even as $e$ goes to $0$,
so $b = 2a$ and the value is at the midpoint.
In a more modern approach, this would be the same as looking at this expression:
```math
\frac{f(x+e) - f(x)}{e} = 0.
```
Working on the left hand side, for non-zero $e$ we can cancel the
common $e$ terms, and then let $e$ become $0$. This becomes a problem
in solving $f'(x)=0$. Fermat could compute the derivative for any
polynomial by taking a limit, a task we would do now by the power
rule and the sum and difference of function rules.
This insight holds for other types of functions:
> If $f(c)$ is a relative maximum then either $f'(c) = 0$ or the
> derivative at $c$ does not exist.
When the derivative exists, this says the tangent line is flat. (If it
had a slope, then the the function would increase by moving left or
right, as appropriate, a point we pursue later.)
For a continuous function $f(x)$, call a point $c$ in the domain of
$f$ where either $f'(c)=0$ or the derivative does not exist a **critical**
**point**.
We can combine Bolzano's extreme value theorem with Fermat's insight to get the following:
> A continuous function on $[a,b]$ has an absolute maximum that occurs
> at a critical point $c$, $a < c < b$, or an endpoint, $a$ or $b$.
A similar statement holds for an absolute minimum. This gives a
restricted set of places to look for absolute maximum and minimum values - all the critical points and the endpoints.
It is also the case that all relative extrema occur at a critical point, *however* not all critical points correspond to relative extrema. We will see *derivative tests* that help characterize when that occurs.
```julia;hold=true; echo=false;
### {{{lhopital_32}}}
imgfile = "figures/lhopital-32.png"
caption = L"""
Image number ``32`` from L'Hopitals calculus book (the first) showing that
at a relative minimum, the tangent line is parallel to the
$x$-axis. This of course is true when the tangent line is well defined
by Fermat's observation.
"""
ImageFile(:derivatives, imgfile, caption)
```
### Numeric derivatives
The `ForwardDiff` package provides a means to numerically compute derivatives without approximations at a point. In `CalculusWithJulia` this is extended to find derivatives of functions and the `'` notation is overloaded for function objects. Hence these two give nearly identical answers, the difference being only the type of number used:
```julia;hold=true
f(x) = 3x^3 - 2x
fp(x) = 9x^2 - 2
f'(3), fp(3)
```
##### Example
For the function $f(x) = x^2 \cdot e^{-x}$ find the absolute maximum over the interval $[0, 5]$.
We have that $f(x)$ is continuous on the closed interval of the
question, and in fact differentiable on $(0,5)$, so any critical point
will be a zero of the derivative. We can check for these with:
```julia;
f(x) = x^2 * exp(-x)
cps = find_zeros(f', -1, 6) # find_zeros in `Roots`
```
We get $0$ and $2$ are critical points. The endpoints are $0$ and
$5$. So the absolute maximum over this interval is either at $0$, $2$,
or $5$:
```julia;
f(0), f(2), f(5)
```
We see that $f(2)$ is then the maximum.
A few things. First, `find_zeros` can miss some roots, in particular
endpoints and roots that just touch $0$. We should graph to verify it
didn't. Second, it can be easier sometimes to check the values using
the "dot" notation. If `f`, `a`,`b` are the function and the interval,
then this would typically follow this pattern:
```julia
a, b = 0, 5
critical_pts = find_zeros(f', a, b)
f.(critical_pts), f(a), f(b)
```
For this problem, we have the left endpoint repeated, but in general
this won't be a point where the derivative is zero.
As an aside, the output above is not a single container. To achieve that, the values can be combined before the broadcasting:
```julia
f.(vcat(a, critical_pts, b))
```
##### Example
For the function $g(x) = e^x\cdot(x^3 - x)$ find the absolute maximum over the interval $[0, 2]$.
We follow the same pattern. Since $f(x)$ is continuous on the closed interval and differentiable on the open interval we know that the absolute maximum must occur at an endpoint ($0$ or $2$) or a critical point where $f'(c)=0$. To solve for these, we have again:
```julia;
g(x) = exp(x) * (x^3 - x)
gcps = find_zeros(g', 0, 2)
```
And checking values gives:
```julia;
g.(vcat(0, gcps, 2))
```
Here the maximum occurs at an endpoint. The critical point $c=0.67\dots$
does not produce a maximum value. Rather $f(0.67\dots)$ is an absolute
minimum.
```julia; echo=false
note(L"""
**Absolute minimum** We haven't discussed the parallel problem of
absolute minima over a closed interval. By considering the function
$h(x) = - f(x)$, we see that the any thing true for an absolute
maximum should hold in a related manner for an absolute minimum, in
particular an absolute minimum on a closed interval will only occur
at a critical point or an end point.
""")
```
## Rolle's theorem
Let $f(x)$ be differentiable on $(a,b)$ and continuous on
$[a,b]$. Then the absolute maximum occurs at an endpoint or where the
derivative is ``0`` (as the derivative is always defined). This gives rise to:
> *[Rolle's](http://en.wikipedia.org/wiki/Rolle%27s_theorem) theorem*: For $f$ differentiable on ``(a,b)`` and continuous on ``[a,b]``, if $f(a)=f(b)$, then there exists some $c$ in $(a,b)$ with $f'(c) = 0$.
This modest observation opens the door to many relationships between a function and its derivative, as it ties the two together in one statement.
To see why Rolle's theorem is true, we assume that $f(a)=0$, otherwise
consider $g(x)=f(x)-f(a)$. By the extreme value theorem, there must be
an absolute maximum and minimum. If $f(x)$ is ever positive, then the
absolute maximum occurs in $(a,b)$ - not at an endpoint - so at a
critical point where the derivative is $0$. Similarly if $f(x)$ is
ever negative. Finally, if $f(x)$ is just $0$, then take any $c$ in
$(a,b)$.
The statement in Rolle's theorem speaks to existence. It doesn't give
a recipe to find $c$. It just guarantees that there is *one* or *more*
values in the interval $(a,b)$ where the derivative is $0$ if we
assume differentiability on $(a,b)$ and continuity on $[a,b]$.
##### Example
Let $j(x) = e^x \cdot x \cdot (x-1)$. We know $j(0)=0$ and $j(1)=0$,
so on $[0,1]$. Rolle's theorem
guarantees that we can find *at* *least* one answer (unless numeric
issues arise):
```julia;
j(x) = exp(x) * x * (x-1)
find_zeros(j', 0, 1)
```
This graph illustrates the lone value for $c$ for this problem
```julia; echo=false
x0 = find_zero(j', (0, 1))
plot([j, x->j(x0) + 0*(x-x0)], 0, 1)
```
## The mean value theorem
We are driving south and in one hour cover 70 miles. If the speed
limit is 65 miles per hour, were we ever speeding? We'll we averaged
more than the speed limit so we know the answer is yes, but why?
Speeding would mean our instantaneous speed was more than the speed
limit, yet we only know for sure our *average* speed was more than the
speed limit. The mean value tells us that if some conditions are met,
then at some point (possibly more than one) we must have that our
instantaneous speed is equal to our average speed.
The mean value theorem is a direct generalization of Rolle's theorem.
> *Mean value theorem*: Let $f(x)$ be differentiable on $(a,b)$ and
> continuous on $[a,b]$. Then there exists a value $c$ in $(a,b)$
> where $f'(c) = (f(b) - f(a)) / (b - a)$.
This says for any secant line between $a < b$ there will
be a parallel tangent line at some $c$ with $a < c < b$ (all provided $f$
is differentiable on $(a,b)$ and continuous on $[a,b]$).
This graph illustrates the theorem. The orange line is the secant
line. A parallel line tangent to the graph is guaranteed by the mean
value theorem. In this figure, there are two such lines, rendered
using red.
```julia; hold=true; echo=false
f(x) = x^3 - x
a, b = -2, 1.75
m = (f(b) - f(a)) / (b-a)
cps = find_zeros(x -> f'(x) - m, a, b)
p = plot(f, a-1, b+1, linewidth=3, legend=false)
plot!(x -> f(a) + m*(x-a), a-1, b+1, linewidth=3, color=:orange)
scatter!([a,b], [f(a), f(b)])
for cp in cps
plot!(x -> f(cp) + f'(cp)*(x-cp), a-1, b+1, color=:red)
end
p
```
Like Rolle's theorem this is a guarantee that something exists, not a
recipe to find it. In fact, the mean value theorem is just Rolle's
theorem applied to:
```math
g(x) = f(x) - (f(a) + (f(b) - f(a)) / (b-a) \cdot (x-a))
```
That is the function $f(x)$, minus the secant line between $(a,f(a))$ and $(b, f(b))$.
```julia; hold=true; echo=false
# Need to bring jsxgraph into PLUTO
#caption = """
#Illustration of the mean value theorem from
#[jsxgraph](https://jsxgraph.uni-bayreuth.de/).
#The polynomial function interpolates the points ``A``,``B``,``C``, and ``D``.
#Adjusting these creates different functions. Regardless of the
#function -- which as a polynomial will always be continuous and
#differentiable -- the slope of the secant line between ``A`` and ``B`` is alway#s matched by **some** tangent line between the points ``A`` and ``B``.
#"""
#JSXGraph(:derivatives, "mean-value.js", caption)
nothing
```
An interactive example can be found at [jsxgraph](http://jsxgraph.uni-bayreuth.de/wiki/index.php?title=Mean_Value_Theorem).
##### Example
The mean value theorem is an extremely useful tool to relate properties of a function with properties of its derivative, as, like Rolle's theorem, it includes both ``f`` and ``f'`` in its statement.
For example, suppose we have a function $f(x)$ and we know that the
derivative is **always** $0$. What can we say about the function?
Well, constant functions have derivatives that are constantly $0$.
But do others? We will see the answer is no: If a function has a zero derivative in ``(a,b)`` it must be a constant. We can readily see that if ``f`` is a polynomial function this is the case, as we can differentiate a polynomial function and this will be zero only if **all** its coefficients are ``0``, which would mean there is no non-constant leading term in the polynomial. But polynomials are not representative of all functions, and so a proof requires a bit more effort.
Suppose it is known that $f'(x)=0$ on some interval ``I`` and we take any ``a < b`` in ``I``. Since $f'(x)$ always exists, $f(x)$ is always differentiable, and
hence always continuous. So on $[a,b]$ the conditions of the mean
value theorem apply. That is, there is a $c$ in ``(a,b)`` with $(f(b) - f(a)) / (b-a) =
f'(c) = 0$. But this would imply $f(b) - f(a)=0$. That is $f(x)$ is a
constant, as for any $a$ and $b$, we see $f(a)=f(b)$.
### The Cauchy mean value theorem
[Cauchy](http://en.wikipedia.org/wiki/Mean_value_theorem#Cauchy.27s_mean_value_theorem)
offered an extension to the mean value theorem above. Suppose both $f$
and $g$ satisfy the conditions of the mean value theorem on $[a,b]$ with $g(b)-g(a) \neq 0$,
then there exists at least one $c$ with $a < c < b$ such that
```math
f'(c) = g'(c) \cdot \frac{f(b) - f(a)}{g(b) - g(a)}.
```
The proof follows by considering $h(x) = f(x) - r\cdot g(x)$, with $r$ chosen so that $h(a)=h(b)$. Then Rolle's theorem applies so that there is a $c$ with $h'(c)=0$, so $f'(c) = r g'(c)$, but $r$ can be seen to be $(f(b)-f(a))/(g(b)-g(a))$, which proves the theorem.
Letting $g(x) = x$ demonstrates that the mean value theorem is a special case.
##### Example
Suppose $f(x)$ and $g(x)$ satisfy the Cauchy mean value theorem on
$[0,x]$, $g'(x)$ is non-zero on $(0,x)$, and $f(0)=g(0)=0$. Then we have:
```math
\frac{f(x) - f(0)}{g(x) - g(0)} = \frac{f(x)}{g(x)} = \frac{f'(c)}{g'(c)},
```
For some $c$ in $[0,x]$. If $\lim_{x \rightarrow 0} f'(x)/g'(x) = L$,
then the right hand side will have a limit of $L$, and hence the left
hand side will too. That is, when the limit exists, we have under
these conditions that $\lim_{x\rightarrow 0}f(x)/g(x) =
\lim_{x\rightarrow 0}f'(x)/g'(x)$.
This could be used to prove the limit of $\sin(x)/x$ as $x$ goes to
$0$ just by showing the limit of $\cos(x)/1$ is $1$, as is known by
continuity.
### Visualizing the Cauchy mean value theorem
The Cauchy mean value theorem can be visualized in terms of a tangent
line and a *parallel* secant line in a similar manner as the mean
value theorem as long as a *parametric* graph is used. A parametric
graph plots the points $(g(t), f(t))$ for some range of $t$. That is,
it graphs *both* functions at the same time. The following illustrates
the construction of such a graph:
```julia; hold=true; echo=false; cache=true
### {{{parametric_fns}}}
function parametric_fns_graph(n)
f = (x) -> sin(x)
g = (x) -> x
ns = (1:10)/10
ts = range(-pi/2, stop=-pi/2 + ns[n] * pi, length=100)
plt = plot(f, g, -pi/2, -pi/2 + ns[n] * pi, legend=false, size=fig_size,
xlim=(-1.1,1.1), ylim=(-pi/2-.1, pi/2+.1))
scatter!(plt, [f(ts[end])], [g(ts[end])], color=:orange, markersize=5)
val = @sprintf("% 0.2f", ts[end])
annotate!(plt, [(0, 1, "t = $val")])
end
caption = L"""
Illustration of parametric graph of $(g(t), f(t))$ for $-\pi/2 \leq t
\leq \pi/2$ with $g(x) = \sin(x)$ and $f(x) = x$. Each point on the
graph is from some value $t$ in the interval. We can see that the
graph goes through $(0,0)$ as that is when $t=0$. As well, it must go
through $(1, \pi/2)$ as that is when $t=\pi/2$
"""
n = 10
anim = @animate for i=1:n
parametric_fns_graph(i)
end
imgfile = tempname() * ".gif"
gif(anim, imgfile, fps = 1)
ImageFile(imgfile, caption)
```
With $g(x) = \sin(x)$ and $f(x) = x$, we can take $I=[a,b] =
[0, \pi/2]$. In the figure below, the *secant line* is drawn in red which
connects $(g(a), f(a))$ with the point $(g(b), f(b))$, and hence
has slope $\Delta f/\Delta g$. The parallel lines drawn show the *tangent* lines with slope $f'(c)/g'(c)$. Two exist for this problem, the mean value theorem guarantees at least one will.
```julia; hold=true; echo=false
g(x) = sin(x)
f(x) = x
ts = range(-pi/2, stop=pi/2, length=50)
a,b = 0, pi/2
m = (f(b) - f(a))/(g(b) - g(a))
cps = find_zeros(x -> f'(x)/g'(x) - m, -pi/2, pi/2)
c = cps[1]
Delta = (0 + m * (c - 0)) - (g(c))
p = plot(g, f, -pi/2, pi/2, linewidth=3, legend=false)
plot!(x -> f(a) + m * (x - g(a)), -1, 1, linewidth=3, color=:red)
scatter!([g(a),g(b)], [f(a), f(b)])
for c in cps
plot!(x -> f(c) + m * (x - g(c)), -1, 1, color=:orange)
end
p
```
## Questions
###### Question
Rolle's theorem is a guarantee of a value, but does not provide a recipe to find it. For the function $1 - x^2$ over the interval $[-5,5]$, find a value $c$ that satisfies the result.
```julia; hold=true; echo=false
c = 0
numericq(c)
```
###### Question
The extreme value theorem is a guarantee of a value, but does not provide a recipe to find it. For the function $f(x) = \sin(x)$ on $I=[0, \pi]$ find a value $c$ satisfying the theorem for an absolute maximum.
```julia; hold=true; echo=false
c = pi/2
numericq(c)
```
###### Question
The extreme value theorem is a guarantee of a value, but does not provide a recipe to find it. For the function $f(x) = \sin(x)$ on $I=[\pi, 3\pi/2]$ find a value $c$ satisfying the theorem for an absolute maximum.
```julia; hold=true; echo=false
c = pi
numericq(c)
```
###### Question
The mean value theorem is a guarantee of a value, but does not provide a recipe to find it. For $f(x) = x^2$ on $[0,2]$ find a value of $c$ satisfying the theorem.
```julia; hold=true; echo=false
c = 1
numericq(c)
```
###### Question
The Cauchy mean value theorem is a guarantee of a value, but does not provide a recipe to find it. For $f(x) = x^3$ and $g(x) = x^2$ find a value $c$ in the interval $[1, 2]$
```julia; hold=true; echo=false
c,x = symbols("c, x", real=true)
val = solve(3c^2 / (2c) - (2^3 - 1^3) / (2^2 - 1^2), c)[1]
numericq(float(val))
```
###### Question
Will the function $f(x) = x + 1/x$ satisfy the conditions of the mean value theorem over $[-1/2, 1/2]$?
```julia; hold=true; echo=false
radioq(["Yes", "No"], 2)
```
###### Question
Just as it is a fact that $f'(x) = 0$ (for all $x$ in $I$) implies
$f(x)$ is a constant, so too is it a fact that if $f'(x) = g'(x)$ that
$f(x) - g(x)$ is a constant. What function would you consider, if you
wanted to prove this with the mean value theorem?
```julia; hold=true; echo=false
choices = [
"``h(x) = f(x) - (f(b) - f(a)) / (b - a)``",
"``h(x) = f(x) - (f(b) - f(a)) / (b - a) \\cdot g(x)``",
"``h(x) = f(x) - g(x)``",
"``h(x) = f'(x) - g'(x)``"
]
ans = 3
radioq(choices, ans)
```
###### Question
Suppose $f''(x) > 0$ on $I$. Why is it impossible that $f'(x) = 0$ at more than one value in $I$?
```julia; hold=true; echo=false
choices = [
L"It isn't. The function $f(x) = x^2$ has two zeros and $f''(x) = 2 > 0$",
"By the Rolle's theorem, there is at least one, and perhaps more",
L"By the mean value theorem, we must have $f'(b) - f'(a) > 0$ when ever $b > a$. This means $f'(x)$ is increasing and can't double back to have more than one zero."
]
ans = 3
radioq(choices, ans)
```
###### Question
Let $f(x) = 1/x$. For $0 < a < b$, find $c$ so that $f'(c) = (f(b) - f(a)) / (b-a)$.
```julia; hold=true; echo=false
choices = [
"``c = (a+b)/2``",
"``c = \\sqrt{ab}``",
"``c = 1 / (1/a + 1/b)``",
"``c = a + (\\sqrt{5} - 1)/2 \\cdot (b-a)``"
]
ans = 2
radioq(choices, ans)
```
###### Question
Let $f(x) = x^2$. For $0 < a < b$, find $c$ so that $f'(c) = (f(b) - f(a)) / (b-a)$.
```julia; hold=true; echo=false
choices = [
"``c = (a+b)/2``",
"``c = \\sqrt{ab}``",
"``c = 1 / (1/a + 1/b)``",
"``c = a + (\\sqrt{5} - 1)/2 \\cdot (b-a)``"
]
ans = 1
radioq(choices, ans)
```
###### Question
In an example, we used the fact that if $0 < c < x$, for some $c$ given by the mean value theorem and $f(x)$ goes to $0$ as $x$ goes to zero then $f(c)$ will also go to zero. Suppose we say that $c=g(x)$ for some function $c$.
Why is it known that $g(x)$ goes to $0$ as $x$ goes to zero (from the right)?
```julia; hold=true; echo=false
choices = [L"The squeeze theorem applies, as $0 < g(x) < x$.",
L"As $f(x)$ goes to zero by Rolle's theorem it must be that $g(x)$ goes to $0$.",
L"This follows by the extreme value theorem, as there must be some $c$ in $[0,x]$."]
ans = 1
radioq(choices, ans)
```
Since $g(x)$ goes to zero, why is it true that if $f(x)$ goes to $L$ as $x$ goes to zero that $f(g(x))$ must also have a limit $L$?
```julia; hold=true; echo=false
choices = ["It isn't true. The limit must be 0",
L"The squeeze theorem applies, as $0 < g(x) < x$",
"This follows from the limit rules for composition of functions"]
ans = 3
radioq(choices, ans)
```

530
CwJ/derivatives/more_zeros.jmd Normal file
View File

@@ -0,0 +1,530 @@
# Derivative-free alternatives to Newton's method
This section uses these add-on packages:
```julia
using CalculusWithJulia
using Plots
using ImplicitEquations
using Roots
using SymPy
```
```julia; echo=false; results="hidden"
using CalculusWithJulia.WeaveSupport
const frontmatter = (
title = "Derivative-free alternatives to Newton's method",
description = "Calculus with Julia: Derivative-free alternatives to Newton's method",
tags = ["CalculusWithJulia", "derivatives", "derivative-free alternatives to newton's method"],
);
nothing
```
----
Newton's method is not the only algorithm of its kind for identifying zeros of a function. In this section we discuss some alternatives.
## The `find_zero(f, x0)` function
The function `find_zero` from the `Roots` packages provides several different algorithms for finding a zero of a function, including some a derivative-free
algorithms for finding zeros when started with an initial
guess. The default method is similar to Newton's method in that only a good initial
guess is needed. However, the algorithm, while possibly slower in terms of
function evaluations and steps, is engineered to be a bit more
robust to the choice of initial estimate than Newton's method. (If it
finds a bracket, it will use a bisection algorithm which is guaranteed to
converge, but can be slower to do so.) Here we see how to call the
function:
```julia;
f(x) = cos(x) - x
x₀ = 1
find_zero(f, x₀)
```
Compare to this related call which uses the bisection method:
```julia;
find_zero(f, (0, 1)) ## [0,1] must be a bracketing interval
```
For this example both give the same answer, but the bisection method
is a bit less convenient as a bracketing interval must be pre-specified.
## The secant method
The default `find_zero` method above uses a secant-like method unless a bracketing method is found. The secant method is historic, dating back over ``3000`` years. Here we discuss the secant method in a more general framework.
One way to view Newton's method is through the inverse of ``f`` (assuming it exists): if ``f(\alpha) = 0`` then ``\alpha = f^{-1}(0)``.
If ``f`` has a simple zero at ``\alpha`` and is locally invertible (that is some ``f^{-1}`` exists) then the update step for Newton's method can be identified with:
* fitting a polynomial to the local inverse function of ``f`` going through through the point ``(f(x_0),x_0)``,
* and matching the slope of ``f`` at the same point.
That is, we can write ``g(y) = h_0 + h_1 (y-f(x_0))``. Then ``g(f(x_0)) = x_0 = h_0``, so ``h_0 = x_0``. From ``g'(f(x_0)) = 1/f'(x_0)``, we get ``h_1 = 1/f'(x_0)``. That is, ``g(y) = x_0 + (y-f(x_0))/f'(x_0)``. At ``y=0,`` we get the update step ``x_1 = g(0) = x_0 - f(x_0)/f'(x_0)``.
A similar viewpoint can be used to create derivative-free methods.
For example, the [secant method](https://en.wikipedia.org/wiki/Secant_method) can be seen as the result of fitting a degree-``1`` polynomial approximation for ``f^{-1}`` through two points ``(f(x_0),x_0)`` and ``(f(x_1), x_1)``.
Again, expressing this approximation as ``g(y) = h_0 + h_1(y-f(x_1))`` leads to ``g(f(x_1)) = x_1 = h_0``.
Substituting ``f(x_0)`` gives ``g(f(x_0)) = x_0 = x_1 + h_1(f(x_0)-f(x_1))``. Solving for ``h_1`` leads to ``h_1=(x_1-x_0)/(f(x_1)-f(x_0))``. Then ``x_2 = g(0) = x_1 + (x_1-x_0)/(f(x_1)-f(x_0)) \cdot f(x_1)``. This is the first step of the secant method:
```math
x_{n+1} = x_n - f(x_n) \frac{x_n - x_{n-1}}{f(x_n) - f(x_{n-1})}.
```
That is, where the next step of Newton's method comes from the intersection of the tangent line at ``x_n`` with the ``x``-axis, the next step of the secant method comes from the intersection of the secant line defined by ``x_n`` and ``x_{n-1}`` with the ``x`` axis. That is, the secant method simply replaces ``f'(x_n)`` with the slope of the secant line between ``x_n`` and ``x_{n-1}``.
We code the update step as `λ2`:
```julia;
λ2(f0,f1,x0,x1) = x1 - f1 * (x1-x0) / (f1-f0)
```
Then we can run a few steps to identify the zero of sine starting at ``3`` and ``4``
```julia; hold=true; term=true
x0,x1 = 4,3
f0,f1 = sin.((x0,x1))
@show x1,f1
x0,x1 = x1, λ2(f0,f1,x0,x1)
f0,f1 = f1, sin(x1)
@show x1,f1
x0,x1 = x1, λ2(f0,f1,x0,x1)
f0,f1 = f1, sin(x1)
@show x1,f1
x0,x1 = x1, λ2(f0,f1,x0,x1)
f0,f1 = f1, sin(x1)
@show x1,f1
x0,x1 = x1, λ2(f0,f1,x0,x1)
f0,f1 = f1, sin(x1)
x1,f1
```
Like Newton's method, the secant method coverges quickly for this problem (though its rate is less than the quadratic rate of Newton's method).
This method is included in `Roots` as `Secant()` (or `Order1()`):
```julia;
find_zero(sin, (4,3), Secant())
```
Though the derivative is related to the slope of the secant line, that is in the limit. The convergence of the secant method is not as fast as Newton's method, though at each step of the secant method, only one new function evaluation is needed, so it can be more efficient for functions that are expensive to compute or differentiate.
Let ``\epsilon_{n+1} = x_{n+1}-\alpha``, where ``\alpha`` is assumed to be the *simple* zero of ``f(x)`` that the secant method converges to. A [calculation](https://math.okstate.edu/people/binegar/4513-F98/4513-l08.pdf) shows that
```math
\begin{align*}
\epsilon_{n+1} &\approx \frac{x_n-x_{n-1}}{f(x_n)-f(x_{n-1})} \frac{(1/2)f''(\alpha)(e_n-e_{n-1})}{x_n-x_{n-1}} \epsilon_n \epsilon_{n-1}\\
& \approx \frac{f''(\alpha)}{2f'(\alpha)} \epsilon_n \epsilon_{n-1}\\
&= C \epsilon_n \epsilon_{n-1}.
\end{align*}
```
The constant `C` is similar to that for Newton's method, and reveals potential troubles for the secant method similar to those of Newton's method: a poor initial guess (the initial error is too big), the second derivative is too large, the first derivative too flat near the answer.
Assuming the error term has the form ``\epsilon_{n+1} = A|\epsilon_n|^\phi`` and substituting into the above leads to the equation
```math
\frac{A^{1-1/\phi}}{C} = |\epsilon_n|^{1 - \phi +1/\phi}.
```
The left side being a constant suggests ``\phi`` solves: ``1 - \phi + 1/\phi = 0`` or ``\phi^2 -\phi - 1 = 0``. The solution is the golden ratio, ``(1 + \sqrt{5})/2 \approx 1.618\dots``.
### Steffensen's method
Steffensen's method is a secant-like method that converges with ``|\epsilon_{n+1}| \approx C |\epsilon_n|^2``. The secant is taken between the points ``(x_n,f(x_n))`` and ``(x_n + f(x_n), f(x_n + f(x_n))``. Like Newton's method this requires ``2`` function evaluations per step. Steffensen's is implemented through `Roots.Steffensen()`. Steffensen's method is more sensitive to the initial guess than other methods, so in practice must be used with care, though it is a starting point for many higher-order derivative-free methods.
## Inverse quadratic interpolation
Inverse quadratic interpolation fits a quadratic polynomial through three points, not just two like the Secant method. The third being ``(f(x_2), x_2)``.
For example, here is the inverse quadratic function, ``g(y)``, going through three points marked with red dots. The blue dot is found from ``(g(0), 0)``.
```julia; hold=true; echo=false
a,b,c = 1,2,3
fa,fb,fc = -1,1/4,1
g(y) = (y-fb)*(y-fa)/(fc-fb)/(fc-fa)*c + (y-fc)*(y-fa)/(fb-fc)/(fb-fa)*b + (y-fc)*(y-fb)/(fa-fc)/(fa-fb)*a
ys = range(-2,2, length=100)
xs = g.(ys)
plot(xs, ys, legend=false)
scatter!([a,b,c],[fa,fb,fc], color=:red, markersize=5)
scatter!([g(0)],[0], color=:blue, markersize=5)
plot!(zero, color=:blue)
```
Here we use `SymPy` to identify the degree-``2`` polynomial as a function of ``y``, then evaluate it at ``y=0`` to find the next step:
```julia
@syms y hs[0:2] xs[0:2] fs[0:2]
H(y) = sum(hᵢ*(y - fs[end])^i for (hᵢ,i) ∈ zip(hs, 0:2))
eqs = [H(fᵢ) ~ xᵢ for (xᵢ, fᵢ) ∈ zip(xs, fs)]
ϕ = solve(eqs, hs)
hy = subs(H(y), ϕ)
```
The value of `hy` at ``y=0`` yields the next guess based on the past three, and is given by:
```julia;
q⁻¹ = hy(y => 0)
```
Though the above can be simplified quite a bit when computed by hand, here we simply make this a function with `lambdify` which we will use below.
```julia;
λ3 = lambdify(q⁻¹) # fs, then xs
```
(`SymPy`'s `lambdify` function, by default, picks the order of its argument lexicographically, in this case they will be the `f` values then the `x` values.)
An inverse quadratic step is utilized by Brent's method, as possible, to yield a rapidly convergent bracketing algorithm implemented as a default zero finder in many software languages. `Julia`'s `Roots` package implements the method in `Roots.Brent()`. An inverse cubic interpolation is utilized by [Alefeld, Potra, and Shi](https://dl.acm.org/doi/10.1145/210089.210111) which gives an asymptotically even more rapidly convergent algorithm then Brent's (implemented in `Roots.AlefeldPotraShi()` and also `Roots.A42()`). This is used as a finishing step in many cases by the default hybrid `Order0()` method of `find_zero`.
In a bracketing algorithm, the next step should reduce the size of the bracket, so the next iterate should be inside the current bracket. However, quadratic convergence does not guarantee this to happen. As such, sometimes a subsitute method must be chosen.
[Chandrapatla's](https://www.google.com/books/edition/Computational_Physics/cC-8BAAAQBAJ?hl=en&gbpv=1&pg=PA95&printsec=frontcover) method, is a bracketing method utilizing an inverse quadratic step as the centerpiece. The key insight is the test to choose between this inverse quadratic step and a bisection step. This is done in the following based on values of ``\xi`` and ``\Phi`` defined within:
```julia;
function chandrapatla(f, u, v, λ; verbose=false)
a,b = promote(float(u), float(v))
fa,fb = f(a),f(b)
@assert fa * fb < 0
if abs(fa) < abs(fb)
a,b,fa,fb = b,a,fb,fa
end
c, fc = a, fa
maxsteps = 100
for ns in 1:maxsteps
Δ = abs(b-a)
m, fm = (abs(fa) < abs(fb)) ? (a, fa) : (b, fb)
ϵ = eps(m)
if Δ ≤ 2ϵ
return m
end
@show m,fm
iszero(fm) && return m
ξ = (a-b)/(c-b)
Φ = (fa-fb)/(fc-fb)
if Φ^2 < ξ < 1 - (1-Φ)^2
xt = λ(fa,fc,fb, a,c,b) # inverse quadratic
else
xt = a + (b-a)/2
end
ft = f(xt)
isnan(ft) && break
if sign(fa) == sign(ft)
c,fc = a,fa
a,fa = xt,ft
else
c,b,a = b,a,xt
fc,fb,fa = fb,fa,ft
end
verbose && @show ns, a, fa
end
error("no convergence: [a,b] = $(sort([a,b]))")
end
```
Like bisection, this method ensures that ``a`` and ``b`` is a bracket, but it moves ``a`` to the newest estimate, so does not maintain that ``a < b`` throughout.
We can see it in action on the sine function. Here we pass in ``\lambda``, but in a real implementation (as in `Roots.Chandrapatla()`) we would have programmed the algorithm to compute the inverse quadratic value.
```julia; term=true
chandrapatla(sin, 3, 4, λ3, verbose=true)
```
The condition `Φ^2 < ξ < 1 - (1-Φ)^2` can be visualized. Assume `a,b=0,1`, `fa,fb=-1/2,1`, Then `c < a < b`, and `fc` has the same sign as `fa`, but what values of `fc` will satisfy the inequality?
```julia;
ξ(c,fc) = (a-b)/(c-b)
Φ(c,fc) = (fa-fb)/(fc-fb)
Φl(c,fc) = Φ(c,fc)^2
Φr(c,fc) = 1 - (1-Φ(c,fc))^2
a,b = 0, 1
fa,fb = -1/2, 1
region = Lt(Φl, ξ) & Lt(ξ,Φr)
plot(region, xlims=(-2,a), ylims=(-3,0))
```
When `(c,fc)` is in the shaded area, the inverse quadratic step is chosen. We can see that `fc < fa` is needed.
For these values, this area is within the area where a implicit quadratic step will result in a value between `a` and `b`:
```julia;
l(c,fc) = λ3(fa,fb,fc,a,b,c)
region₃ = ImplicitEquations.Lt(l,b) & ImplicitEquations.Gt(l,a)
plot(region₃, xlims=(-2,0), ylims=(-3,0))
```
There are values in the parameter space where this does not occur.
## Tolerances
The `chandrapatla` algorithm typically waits until `abs(b-a) <= 2eps(m)` (where ``m`` is either ``b`` or ``a`` depending on the size of ``f(a)`` and ``f(b)``) is satisfied. Informally this means the algorithm stops when the two bracketing values are no more than a small amount apart. What is a "small amount?"
To understand, we start with the fact that floating point numbers are an approximation to real numbers.
Floating point numbers effectively represent a number in scientific
notation in terms of
* a sign (plus or minus) ,
* a *mantissa* (a number in ``[1,2)``, in binary ), and
* an exponent (to represent a power of ``2``).
The mantissa is of the form `1.xxxxx...xxx` where there are ``m``
different `x`s each possibly a `0` or `1`. The `i`th `x` indicates if the term `1/2^i` should be
included in the value. The mantissa is the sum of `1` plus the
indicated values of `1/2^i` for `i` in `1` to `m`. So the last `x` represents if `1/2^m` should be
included in the sum. As such, the
mantissa represents a discrete set of values, separated by `1/2^m`, as
that is the smallest difference possible.
For example if `m=2` then the possible value for the mantissa are `11 => 1 + 1/2 + 1/4 = 7/4`,
`10 => 1 + 1/2 = 6/4`, `01 => 1 + 1/4 = 5/4`. and `00 => 1 = 4/4`, values separated by `1/4 = 1/2^m`.
For ``64``-bit floating point numbers `m=52`, so the values in the mantissa differ by `1/2^52 = 2.220446049250313e-16`. This is the value of `eps()`.
However, this "gap" between numbers is for values when the exponent is `0`. That is the numbers in `[1,2)`. For values in `[2,4)` the gap is twice, between `[1/2,1)` the gap is half. That is the gap depends on the size of the number. The gap between `x` and its next largest floating point number is given by `eps(x)` and that always satisfies `eps(x) <= eps() * abs(x)`.
One way to think about this is the difference between `x` and the next largest floating point values is *basically* `x*(1+eps()) - x` or `x*eps()`.
For the specific example, `abs(b-a) <= 2eps(m)` means that the gap between `a` and `b` is essentially 2 floating point values from the ``x`` value with the smallest ``f(x)`` value.
For bracketing methods that is about as good as you can get. However, once floating values are understood, the absolute best you can get for a bracketing interval would be
* along the way, a value `f(c)` is found which is *exactly* `0.0`
* the endpoints of the bracketing interval are *adjacent* floating point values, meaning the interval can not be bisected and `f` changes sign between the two values.
There can be problems when the stopping criteria is `abs(b-a) <= 2eps(m))` and the answer is `0.0` that require engineering around. For example, the algorithm above for the function `f(x) = -40*x*exp(-x)` does not converge when started with `[-9,1]`, even though `0.0` is an obvious zero.
```julia; hold=true
fu(x) = -40*x*exp(-x)
chandrapatla(fu, -9, 1, λ3)
```
Here the issue is `abs(b-a)` is tiny (of the order `1e-119`) but `eps(m)` is even smaller.
For non-bracketing methods, like Newton's method or the secant method, different criteria are useful.
There may not be a bracketing interval for `f` (for example `f(x) = (x-1)^2`) so the second criteria above might need to be restated in terms of the last two iterates, ``x_n`` and ``x_{n-1}``. Calling this difference ``\Delta = |x_n - x_{n-1}|``, we might stop if ``\Delta`` is small enough. As there are scenarios where this can happen, but the function is not at a zero, a check on the size of ``f`` is needed.
However, there may be no floating point value where ``f`` is exactly `0.0` so checking the size of `f(x_n)` requires some agreement.
First if `f(x_n)` is `0.0` then it makes sense to call `x_n` an *exact zero* of ``f``, even though this may hold even if `x_n`, a floating point value, is not mathematically an *exact* zero of ``f``. (Consider `f(x) = x^2 - 2x + 1`. Mathematically, this is identical to `g(x) = (x-1)^2`, but `f(1 + eps())` is zero, while `g(1+eps())` is `4.930380657631324e-32`.
However, there may never be a value with `f(x_n)` exactly `0.0`. (The value of `sin(pi)` is not zero, for example, as `pi` is an approximation to ``\pi``, as well the `sin` of values adjacent to `float(pi)` do not produce `0.0` exactly.)
Suppose `x_n` is the closest floating number to ``\alpha``, the zero. Then the relative rounding error, ``(`` `x_n` ``- \alpha)/\alpha``, will be a value ``(1 + \delta)`` with ``\delta`` less than `eps()`.
How far then can `f(x_n)` be from ``0 = f(\alpha)``?
```math
f(x_n) = f(x_n - \alpha + \alpha) = f(\alpha + \alpha \cdot \delta) = f(\alpha \cdot (1 + \delta)),
```
Assuming ``f`` has a derivative, the linear approximation gives:
```math
f(x_n) \approx f(\alpha) + f'(\alpha) \cdot (\alpha\delta) = f'(\alpha) \cdot \alpha \delta
```
So we should consider `f(x_n)` an *approximate zero* when it is on the scale of
``f'(\alpha) \cdot \alpha \delta``.
That ``\alpha`` factor means we consider a *relative* tolerance for `f`.
Also important -- when `x_n` is close to `0`,
is the need for an *absolute* tolerance, one not dependent on the size of `x`.
So a good condition to check if `f(x_n)` is small is
`abs(f(x_n)) <= abs(x_n) * rtol + atol`, or `abs(f(x_n)) <= max(abs(x_n) * rtol, atol)`
where the relative tolerance, `rtol`, would absorb an estimate for ``f'(\alpha)``.
Now, in Newton's method the update step is ``f(x_n)/f'(x_n)``. Naturally when ``f(x_n)`` is close to ``0``, the update step is small and ``\Delta`` will be close to ``0``. *However*, should ``f'(x_n)`` be large, then ``\Delta`` can also be small and the algorithm will possibly stop, as ``x_{n+1} \approx x_n`` -- but not necessarily ``x_{n+1} \approx \alpha``. So termination on ``\Delta`` alone can be off. Checking if ``f(x_{n+1})`` is an approximate zero is also useful to include in a stopping criteria.
One thing to keep in mind is that the right-hand side of the rule `abs(f(x_n)) <= abs(x_n) * rtol + atol`, as a function of `x_n`, goes to `Inf` as `x_n` increases. So if `f` has `0` as an asymptote (like `e^(-x)`) for large enough `x_n`, the rule will be `true` and `x_n` could be counted as an approximate zero, despite it not being one.
So a modified criteria for convergence might look like:
* stop if ``\Delta`` is small and `f` is an approximate zero with some tolerances
* stop if `f` is an approximate zero with some tolerances, but be mindful that this rule can identify mathematically erroneous answers.
It is not uncommon to assign `rtol` to have a value like `sqrt(eps())` to account for accumulated floating point errors and the factor of ``f'(\alpha)``, though in the `Roots` package it is set smaller by default.
## Questions
###### Question
Let `f(x) = tanh(x)` (the hyperbolic tangent) and `fp(x) = sech(x)^2`, its derivative.
Does *Newton's* method (using `Roots.Newton()`) converge starting at `1.0`?
```julia; hold=true; echo=false
yesnoq("yes")
```
Does *Newton's* method (using `Roots.Newton()`) converge starting at `1.3`?
```julia; hold=true; echo=false
yesnoq("no")
```
Does the secant method (using `Roots.Secant()`) converge starting at `1.3`? (a second starting value will automatically be chosen, if not directly passed in.)
```julia; hold=true; echo=false
yesnoq("yes")
```
###### Question
For the function `f(x) = x^5 - x - 1` both Newton's method and the secant method will converge to the one root when started from `1.0`. Using `verbose=true` as an argument to `find_zero`, (e.g., `find_zero(f, x0, Roots.Secant(), verbose=true)`) how many *more* steps does the secant method need to converge?
```julia; hold=true; echo=false
numericq(2)
```
Do the two methods converge to the exact same value?
```julia; hold=true; echo=false
yesnoq("yes")
```
###### Question
Let `f(x) = exp(x) - x^4` and `x0=8.0`. How many steps (iterations) does it take for the secant method to converge using the default tolerances?
```julia; hold=true; echo=false
numericq(10, 1)
```
###### Question
Let `f(x) = exp(x) - x^4` and a starting bracket be `x0 = [8.9]`. Then calling `find_zero(f,x0, verbose=true)` will show that 49 steps are needed for exact bisection to converge. What about with the `Roots.Brent()` algorithm, which uses inverse quadratic steps when it can?
It takes how many steps?
```julia; hold=true; echo=false
numericq(36, 1)
```
The `Roots.A42()` method uses inverse cubic interpolation, as possible, how many steps does this method take to converge?
```julia; hold=true; echo=false
numericq(3, 1)
```
The large difference is due to how the tolerances are set within `Roots`. The `Brent method gets pretty close in a few steps, but takes a much longer time to get close enough for the default tolerances
###### Question
Consider this crazy function defined by:
```julia; eval=false
f(x) = cos(100*x)-4*erf(30*x-10)
```
(The `erf` function is the (error function](https://en.wikipedia.org/wiki/Error_function) and is in the `SpecialFunctions` package loaded with `CalculusWithJulia`.)
Make a plot over the interval $[-3,3]$ to see why it is called "crazy".
Does `find_zero` find a zero to this function starting from $0$?
```julia; hold=true; echo=false
yesnoq("yes")
```
If so, what is the value?
```julia; hold=true; echo=false
f(x) = cos(100*x)-4*erf(30*x-10)
val = find_zero(f, 0)
numericq(val)
```
If not, what is the reason?
```julia; hold=true; echo=false
choices = [
"The zero is a simple zero",
"The zero is not a simple zero",
"The function oscillates too much to rely on the tangent line approximation far from the zero",
"We can find an answer"
]
ans = 4
radioq(choices, ans, keep_order=true)
```
Does `find_zero` find a zero to this function starting from $1$?
```julia; hold=true; echo=false
yesnoq(false)
```
If so, what is the value?
```julia; hold=true; echo=false
numericq(-999.999)
```
If not, what is the reason?
```julia; hold=true; echo=false
choices = [
"The zero is a simple zero",
"The zero is not a simple zero",
"The function oscillates too much to rely on the tangent line approximations far from the zero",
"We can find an answer"
]
ans = 3
radioq(choices, ans, keep_order=true)
```

72
CwJ/derivatives/newtons-method.js Normal file
View File

@@ -0,0 +1,72 @@
// newton's method
const b = JXG.JSXGraph.initBoard('jsxgraph', {
boundingbox: [-3,5,3,-5], axis:true
});
var f = function(x) {return x*x*x*x*x - x - 1};
var fp = function(x) { return 4*x*x*x*x - 1};
var x0 = 0.85;
var nm = function(x) { return x - f(x)/fp(x);};
var l = b.create('point', [-1.5,0], {name:'', size:0});
var r = b.create('point', [1.5,0], {name:'', size:0});
var xaxis = b.create('line', [l,r])
var P0 = b.create('glider', [x0,0,xaxis], {name:'x0'});
var P0a = b.create('point', [function() {return P0.X();},
function() {return f(P0.X());}], {name:''});
var P1 = b.create('point', [function() {return nm(P0.X());},
0], {name:''});
var P1a = b.create('point', [function() {return P1.X();},
function() {return f(P1.X());}], {name:''});
var P2 = b.create('point', [function() {return nm(P1.X());},
0], {name:''});
var P2a = b.create('point', [function() {return P2.X();},
function() {return f(P2.X());}], {name:''});
var P3 = b.create('point', [function() {return nm(P2.X());},
0], {name:''});
var P3a = b.create('point', [function() {return P3.X();},
function() {return f(P3.X());}], {name:''});
var P4 = b.create('point', [function() {return nm(P3.X());},
0], {name:''});
var P4a = b.create('point', [function() {return P4.X();},
function() {return f(P4.X());}], {name:''});
var P5 = b.create('point', [function() {return nm(P4.X());},
0], {name:'x5', strokeColor:'black'});
P0a.setAttribute({fixed:true});
P1.setAttribute({fixed:true});
P1a.setAttribute({fixed:true});
P2.setAttribute({fixed:true});
P2a.setAttribute({fixed:true});
P3.setAttribute({fixed:true});
P3a.setAttribute({fixed:true});
P4.setAttribute({fixed:true});
P4a.setAttribute({fixed:true});
P5.setAttribute({fixed:true});
var sc = '#000000';
b.create('segment', [P0,P0a], {strokeColor:sc, strokeWidth:1});
b.create('segment', [P0a, P1], {strokeColor:sc, strokeWidth:1});
b.create('segment', [P1,P1a], {strokeColor:sc, strokeWidth:1});
b.create('segment', [P1a, P2], {strokeColor:sc, strokeWidth:1});
b.create('segment', [P2,P2a], {strokeColor:sc, strokeWidth:1});
b.create('segment', [P2a, P3], {strokeColor:sc, strokeWidth:1});
b.create('segment', [P3,P3a], {strokeColor:sc, strokeWidth:1});
b.create('segment', [P3a, P4], {strokeColor:sc, strokeWidth:1});
b.create('segment', [P4,P4a], {strokeColor:sc, strokeWidth:1});
b.create('segment', [P4a, P5], {strokeColor:sc, strokeWidth:1});
b.create('functiongraph', [f, -1.5, 1.5])

1350
CwJ/derivatives/newtons_method.jmd Normal file
View File

File diff suppressed because it is too large Load Diff

352
CwJ/derivatives/numeric_derivatives.jmd Normal file
View File

@@ -0,0 +1,352 @@
# Numeric derivatives
This section uses these add-on packages:
```julia
using CalculusWithJulia
using Plots
using ForwardDiff
using SymPy
using Roots
```
```julia; echo=false; results="hidden"
using CalculusWithJulia.WeaveSupport
const frontmatter = (
title = "Numeric derivatives",
description = "Calculus with Julia: Numeric derivatives",
tags = ["CalculusWithJulia", "derivatives", "numeric derivatives"],
);
nothing
```
----
`SymPy` returns symbolic derivatives. Up to choices of simplification, these answers match those that would be derived by hand. This is useful when comparing with known answers and for seeing the structure of the answer. However, there are times we just want to work with the answer numerically. For that we have other options within `Julia`. We discuss approximate derivatives and automatic derivatives. The latter will find wide usage in these notes.
### Approximate derivatives
By approximating the limit of the secant line with a value for a small, but positive, $h$, we get an approximation to the derivative. That is
```math
f'(x) \approx \frac{f(x+h) - f(x)}{h}.
```
This is the forward-difference approximation. The central difference approximation looks both ways:
```math
f'(x) \approx \frac{f(x+h) - f(x-h)}{2h}.
```
Though in general they are different, they are both
approximations. The central difference is usually more accurate for the
same size $h$. However, both are susceptible to round-off errors. The
numerator is a subtraction of like-size numbers - a perfect
opportunity to lose precision.
As such there is a balancing act:
* if $h$ is too small the round-off errors are problematic,
* if $h$ is too big, the approximation to the limit is not good.
For the forward
difference $h$ values around $10^{-8}$ are typically good, for the central
difference, values around $10^{-6}$ are typically good.
##### Example
Let's verify that the forward difference isn't too far off.
```julia;
f(x) = exp(-x^2/2)
c = 1
h = 1e-8
fapprox = (f(c+h) - f(c)) / h
```
We can compare to the actual with:
```julia;
@syms x
df = diff(f(x), x)
factual = N(df(c))
abs(factual - fapprox)
```
The error is about ``1`` part in ``100`` million.
The central difference is better here:
```julia; hold=true
h = 1e-6
cdapprox = (f(c+h) - f(c-h)) / (2h)
abs(factual - cdapprox)
```
----
The [FiniteDifferences](https://github.com/JuliaDiff/FiniteDifferences.jl) and [FiniteDiff](https://github.com/JuliaDiff/FiniteDiff.jl) packages provide performant interfaces for differentiation based on finite differences.
### Automatic derivatives
There are some other ways to compute derivatives numerically that give
much more accuracy at the expense of slightly increased computing
time. Automatic differentiation is the general name for a few
different approaches. These approaches promise less complexity - in
some cases - than symbolic derivatives and more accuracy than
approximate derivatives; the accuracy is on the order of
machine precision.
The `ForwardDiff` package provides one of [several](https://juliadiff.org/) ways for `Julia` to compute automatic derivatives. `ForwardDiff` is well suited for functions encountered in these notes, which depend on at most a few variables and output no more than a few values at once.
The `ForwardDiff` package was loaded in this section; in general its features are available when the `CalculusWithJulia` package is loaded, as that package provides a more convenient interface.
The `derivative` function is not exported by `FiniteDiff`, so its usage requires qualification. To illustrate, to find the derivative of $f(x)$ at a *point* we have this syntax:
```julia
ForwardDiff.derivative(f, c) # derivative is qualified by a module name
```
The `CalculusWithJulia` package defines an operator `D` which goes from finding a derivative at a point with `ForwardDiff.derivative` to defining a function which evaluates the derivative at each point. It is defined along the lines of `D(f) = x -> ForwardDiff.derivative(f,x)` in parallel to how the derivative operation for a function is defined mathematically from the definition for its value at a point.
Here we see the error in estimating $f'(1)$:
```julia;
fauto = D(f)(c) # D(f) is a function, D(f)(c) is the function called on c
abs(factual - fauto)
```
In this case, it is exact.
The `D` operator is defined for most all functions in `Julia`, though, like the `diff` operator in `SymPy` there are some for which it won't work.
##### Example
For $f(x) = \sqrt{1 + \sin(\cos(x))}$ compare the difference between the forward derivative with $h=1e-8$ and that computed by `D` at $x=\pi/4$.
The forward derivative is found with:
```julia;
𝒇(x) = sqrt(1 + sin(cos(x)))
𝒄, 𝒉 = pi/4, 1e-8
fwd = (𝒇(𝒄+𝒉) - 𝒇(𝒄))/𝒉
```
That given by `D` is:
```julia;
ds_value = D(𝒇)(𝒄)
ds_value, fwd, ds_value - fwd
```
Finally, `SymPy` gives an exact value we use to compare:
```julia;
𝒇𝒑 = diff(𝒇(x), x)
```
```julia
actual = N(𝒇𝒑(PI/4))
actual - ds_value, actual - fwd
```
#### Convenient notation
`Julia` allows the possibility of extending functions to different
types. Out of the box, the `'` notation is not employed for functions,
but is used for matrices. It is used in postfix position, as with
`A'`. We can define it to do the same thing as `D` for functions and
then, we can evaluate derivatives with the familiar `f'(x)`.
This is done in `CalculusWithJulia` along the lines of `Base.adjoint(f::Function) = D(f)`.
Then, we have, for example:
```julia; hold=true;
f(x) = sin(x)
f'(pi), f''(pi)
```
##### Example
Suppose our task is to find a zero of the second derivative of $k(x) =
e^{-x^2/2}$ in $[0, 10]$, a known bracket. The `D` function takes a second argument to indicate the order of the derivative (e.g., `D(f,2)`), but we use the more familiar notation:
```julia; hold=true
k(x) = exp(-x^2/2)
find_zero(k'', 0..10)
```
We pass in the function object, `k''`, and not the evaluated function.
## Recap on derivatives in Julia
A quick summary for finding derivatives in `Julia`, as there are $3$ different manners:
* Symbolic derivatives are found using `diff` from `SymPy`
* Automatic derivatives are found using the notation `f'` using `ForwardDiff.derivative`
* approximate derivatives at a point, `c`, for a given `h` are found with `(f(c+h)-f(c))/h`.
For example, here all three are computed and compared:
```julia; hold=true
f(x) = exp(-x)*sin(x)
c = pi
h = 1e-8
fp = diff(f(x),x)
fp, fp(c), f'(c), (f(c+h) - f(c))/h
```
```julia;echo=false
note("""
The use of `'` to find derivatives provided by `CalculusWithJulia` is convenient, and used extensively in these notes, but it needs to be noted that it does **not conform** with the generic meaning of `'` within `Julia`'s wider package ecosystem and may cause issue with linear algebra operations; the symbol is meant for the adjoint of a matrix.
""")
```
## Questions
##### Question
Find the derivative using a forward difference approximation of $f(x) = x^x$ at the point $x=2$ using `h=0.1`:
```julia; hold=true; echo=false
f(x) = x^x
c, h = 2, 0.1
val = (f(c+h) - f(c))/h
numericq(val)
```
Using `D` or `f'` find the value using automatic differentiation
```julia; hold=true; echo=false
f(x) = x^x
c = 2
val = f'(c)
numericq(val)
```
##### Question
Mathematically, as the value of `h` in the forward difference gets
smaller the forward difference approximation gets better. On the
computer, this is thwarted by floating point representation issues (in
particular the error in subtracting two like-sized numbers in forming
$f(x+h)-f(x)$.)
For `1e-16` what is the error (in absolute value) in finding the forward difference
approximation for the derivative of $\sin(x)$ at $x=0$?
```julia; hold=true; echo=false
f(x) = sin(x)
h = 1e-16
c = 0
approx = (f(c+h)-f(c))/h
val = abs(cos(0) - approx)
numericq(val)
```
Repeat for $x=\pi/4$:
```julia; hold=true; echo=false
f(x) = sin(x)
h = 1e-16
c = pi/4
approx = (f(c+h)-f(c))/h
val = abs(cos(0) - approx)
numericq(val)
```
###### Question
Let $f(x) = x^x$. Using `D`, find $f'(3)$.
```julia; hold=true; echo=false
f(x) = x^x
val = D(f)(3)
numericq(val)
```
###### Question
Let $f(x) = \lvert 1 - \sqrt{1 + x}\rvert$. Using `D`, find $f'(3)$.
```julia; hold=true; echo=false
f(x) = abs(1 - sqrt(1 + x))
val = D(f)(3)
numericq(val)
```
###### Question
Let $f(x) = e^{\sin(x)}$. Using `D`, find $f'(3)$.
```julia; hold=true; echo=false
f(x) = exp(sin(x))
val = D(f)(3)
numericq(val)
```
###### Question
For `Julia`'s
`airyai` function find a numeric derivative using the
forward difference. For $c=3$ and $h=10^{-8}$ find the forward
difference approximation to $f'(3)$ for the `airyai` function.
```julia; hold=true; echo=false
h = 1e-8
c = 3
val = (airyai(c+h) - airyai(c))/h
numericq(val)
```
###### Question
Find the rate of change with respect to time of the function $f(t)= 64 - 16t^2$ at $t=1$.
```julia; hold=true; echo=false
fp_(t) = -16*2*t
c = 1
numericq(fp_(c))
```
###### Question
Find the rate of change with respect to height, $h$, of $f(h) = 32h^3 - 62 h + 12$ at $h=2$.
```julia; hold=true; echo=false
fp_(h) = 3*32h^2 - 62
c = 2
numericq(fp_(2))
```

36
CwJ/derivatives/optimization-trapezoid.js Normal file
View File

@@ -0,0 +1,36 @@
// inscribe trapezoid
var R = 5;
var Delta = 0.5
const b = JXG.JSXGraph.initBoard('jsxgraph', {
boundingbox: [-R-Delta,R+Delta,R+Delta,-1], axis:true
});
var xax = b.create("segment", [[0,0],[R,0]]);
var P4 = b.create("glider", [R/2,0, xax], {name: "P_4=(r,0)"});
var CL = b.create('point', [function() {return -P4.X()},0], {name:''});
var CR = b.create('point', [function() {return P4.X()},0], {name:''});
var C = b.create('semicircle', [CL,CR]);
var Crestricted = b.create("functiongraph",
[function(x) {
r = P4.X();
y = Math.sqrt(r*r - x*x);
return y;
}, 0, function() {return P4.X()}]);
var P3 = b.create("glider", [
P4.X()/2,
Math.sqrt(P4.X()*P4.X()*(1 - 1/4)),
Crestricted], {name:"P_3=(x,y)"});
var P1 = b.create('point', [function() {return -Math.abs(P4.X());},
function() {return P4.Y();}], {name:'P_1'});
var P2 = b.create('point', [function() {return -P3.X();},
function() {return P3.Y();}], {name:'P_2'});
var poly = b.create('polygon',[P1, P2, P3, P4], { borders:{strokeColor:'black'} });
b.create('text',[-1.5,.25, function(){ return 'Area='+ poly.Area().toFixed(1); }]);

1267
CwJ/derivatives/optimization.jmd Normal file
View File

File diff suppressed because it is too large Load Diff

44
CwJ/derivatives/process.jl Normal file
View File

@@ -0,0 +1,44 @@
using WeavePynb
using CwJWeaveTpl
fnames = [
"derivatives", ## more questions
"numeric_derivatives",
"mean_value_theorem",
"optimization",
"curve_sketching",
"linearization",
"newtons_method",
"lhopitals_rule", ## Okay - -but could beef up questions..
"implicit_differentiation", ## add more questions?
"related_rates",
"taylor_series_polynomials"
]
process_file(nm; cache=:off) = CwJWeaveTpl.mmd(nm * ".jmd", cache=cache)
function process_files(;cache=:user)
for f in fnames
@show f
process_file(f, cache=cache)
end
end
"""
## TODO derivatives
tangent lines intersect at avearge for a parabola
Should we have derivative results: inverse functions, logarithmic differentiation...
"""

778
CwJ/derivatives/related_rates.jmd Normal file
View File

@@ -0,0 +1,778 @@
# Related rates
This section uses these add-on packaages:
```julia
using CalculusWithJulia
using Plots
using Roots
using SymPy
```
```julia; echo=false; results="hidden"
using CalculusWithJulia.WeaveSupport
fig_size=(600, 400)
const frontmatter = (
title = "Related rates",
description = "Calculus with Julia: Related rates",
tags = ["CalculusWithJulia", "derivatives", "related rates"],
);
nothing
```
----
Related rates problems involve two (or more) unknown quantities that
are related through an equation. As the two variables depend on each
other, also so do their rates - change with respect to some variable
which is often time, though exactly how remains to be
discovered. Hence the name "related rates."
#### Examples
The following is a typical "book" problem:
> A screen saver displays the outline of a ``3`` cm by ``2`` cm rectangle and
> then expands the rectangle in such a way that the ``2`` cm side is
> expanding at the rate of ``4`` cm/sec and the proportions of the
> rectangle never change. How fast is the area of the rectangle
> increasing when its dimensions are ``12`` cm by ``8`` cm?
> [Source.](http://oregonstate.edu/instruct/mth251/cq/Stage9/Practice/ratesProblems.html)
```julia; hold=true; echo=false; cache=true
### {{{growing_rects}}}
## Secant line approaches tangent line...
function growing_rects_graph(n)
w = (t) -> 2 + 4t
h = (t) -> 3/2 * w(t)
t = n - 1
w_2 = w(t)/2
h_2 = h(t)/2
w_n = w(5)/2
h_n = h(5)/2
plt = plot(w_2 * [-1, -1, 1, 1, -1], h_2 * [-1, 1, 1, -1, -1], xlim=(-17,17), ylim=(-17,17),
legend=false, size=fig_size)
annotate!(plt, [(-1.5, 1, "Area = $(round(Int, 4*w_2*h_2))")])
plt
end
caption = L"""
As $t$ increases, the size of the rectangle grows. The ratio of width to height is fixed. If we know the rate of change in time for the width ($dw/dt$) and the height ($dh/dt$) can we tell the rate of change of *area* with respect to time ($dA/dt$)?
"""
n=6
anim = @animate for i=1:n
growing_rects_graph(i)
end
imgfile = tempname() * ".gif"
gif(anim, imgfile, fps = 1)
ImageFile(imgfile, caption)
```
Here we know $A = w \cdot h$ and we know some things about how $w$ and
$h$ are related *and* about the rate of how both $w$ and $h$ grow in
time $t$. That means that we could express this growth in terms of
some functions $w(t)$ and $h(t)$, then we can figure out that the area - as a function of $t$ - will be expressed as:
```math
A(t) = w(t) \cdot h(t).
```
We would get by the product rule that the *rate of change* of area with respect to time, $A'(t)$ is just:
```math
A'(t) = w'(t) h(t) + w(t) h'(t).
```
As an aside, it is fairly conventional to suppress the $(t)$ part of
the notation $A=wh$ and to use the Leibniz notation for derivatives:
```math
\frac{dA}{dt} = \frac{dw}{dt} h + w \frac{dh}{dt}.
```
This relationship is true for all $t$, but the problem discusses a
certain value of $t$ - when $w(t)=8$ and $h(t) = 12$. At this same
value of $t$, we have $w'(t) = 4$ and so $h'(t) = 6$. Substituting these 4 values into the 4 unknowns in the formula for $A'(t)$ gives:
```math
A'(t) = 4 \cdot 12 + 8 \cdot 6 = 96.
```
Summarizing, from the relationship between $A$, $w$ and $t$, there is
a relationship between their rates of growth with respect to $t$, a
time variable. Using this and known values, we can compute. In this
case, $A'$ at the specific $t$.
We could also have done this differently. We would recognize the following:
- The area of a rectangle is just:
```julia;
A(w,h) = w * h
```
- The width - expanding at a rate of $4t$ from a starting value of $2$ - must satisfy:
```julia;
w(t) = 2 + 4*t
```
- The height is a constant proportion of the width:
```julia;
h(t) = 3/2 * w(t)
```
This means again that area depends on $t$ through this formula:
```julia;
A(t) = A(w(t), h(t))
```
This is why the rates of change are related: as $w$ and $h$ change in
time, the functional relationship with $A$ means $A$ also changes in
time.
Now to answer the question, when the width is 8, we must have that $t$ is:
```julia;
tstar = find_zero(x -> w(x) - 8, [0, 4]) # or solve by hand to get 3/2
```
The question is to find the rate the area is increasing at the given
time $t$, which is $A'(t)$ or $dA/dt$. We get this by performing the
differentiation, then substituting in the value.
Here we do so with the aid of `Julia`, though this problem could readily be done "by hand."
We have expressed $A$ as a function of $t$ by composition, so can differentiate that:
```julia;
A'(tstar)
```
----
Now what? Why is ``96`` of any interest? It is if the value at a specific
time is needed. But in general, a better question might be to
understand if there is some pattern to the numbers in the figure,
these being $6, 54, 150, 294, 486, 726$. Their differences are the
*average* rate of change:
```julia;
xs = [6, 54, 150, 294, 486, 726]
ds = diff(xs)
```
Those seem to be increasing by a fixed amount each time, which we can see by one more application of `diff`:
```julia;
diff(ds)
```
How can this relationship be summarized? Well, let's go back to what we know, though this time using symbolic math:
```julia;
@syms t
diff(A(t), t)
```
This should be clear: the rate of change, $dA/dt$, is increasing
linearly, hence the second derivative, $dA^2/dt^2$ would be constant,
just as we saw for the average rate of change.
So, for this problem, a constant rate of change in width and height
leads to a linear rate of change in area, put otherwise, linear growth
in both width and height leads to quadratic growth in area.
##### Example
A ladder, with length $l$, is leaning against a wall. We parameterize
this problem so that the top of the ladder is at $(0,h)$ and the
bottom at $(b, 0)$. Then $l^2 = h^2 + b^2$ is a constant.
If the ladder starts to slip away at the base, but remains in contact
with the wall, express the rate of change of $h$ with respect to $t$
in terms of $db/dt$.
We have from implicitly differentiating in $t$ the equation $l^2 = h^2 + b^2$, noting that $l$ is a constant, that:
```math
0 = 2h \frac{dh}{dt} + 2b \frac{db}{dt}.
```
Solving, yields:
```math
\frac{dh}{dt} = -\frac{b}{h} \cdot \frac{db}{dt}.
```
* If $l = 12$ and $db/dt = 2$ when $b=4$, find $dh/dt$.
We just need to find $h$ for this value of $b$, as the other two quantities in the last equation are known.
But $h = \sqrt{l^2 - b^2}$, so the answer is:
```julia;
length, bottom, dbdt = 12, 4, 2
height = sqrt(length^2 - bottom^2)
-bottom/height * dbdt
```
* What happens to the rate as $b$ goes to $l$?
As $b$ goes to $l$, $h$ goes to ``0``, so $b/h$ blows up. Unless $db/dt$
goes to $0$, the expression will become $-\infty$.
##### Example
```julia; hold=true; echo=false
caption = "A man and woman walk towards the light."
imgfile = "figures/long-shadow-noir.png"
ImageFile(:derivatives, imgfile, caption)
```
Shadows are a staple of film noir. In the photo, suppose a man and a woman walk towards a street light. As they approach the light the length of their shadow changes.
Suppose, we focus on the ``5`` foot tall woman. Her shadow comes from a streetlight ``15`` feet high. She is walking at ``3`` feet per second towards the light. What is the rate of change of her shadow?
The setup for this problem involves drawing a right triangle with height ``12`` and base given by the distance ``x`` from the light the woman is *plus* the length ``l`` of the shadow. There is a similar triangle formed by the woman's height with length ``l``. Equating the ratios of the sided gives:
```math
\frac{5}{l} = \frac{12}{x + l}
```
As we need to take derivatives, we work with the reciprocal relationship:
```math
\frac{l}{5} = \frac{x + l}{12}
```
Differentiating in ``t`` gives:
```math
\frac{l'}{5} = \frac{x' + l'}{12}
```
Or
```math
l' \cdot (\frac{1}{5} - \frac{1}{12}) = \frac{x'}{12}
```
Solving for ``l'`` gives an answer in terms of ``x'`` the rate the woman is walking. In this description ``x`` is getting shorter, so ``x'`` would be ``-3`` feet per second and the shadow length would be decreasing at a rate proportional to the walking speed.
##### Example
```julia; hold=true; echo=false
p = plot(; axis=nothing, border=:none, legend=false, aspect_ratio=:equal)
scatter!(p, [0],[50], color=:yellow, markersize=50)
plot!(p, [0, 50], [0,0], linestyle=:dash)
plot!(p, [0,50], [50,0], linestyle=:dot)
plot!(p, [25,25],[25,0], linewidth=5, color=:black)
plot!(p, [25,50], [0,0], linewidth=2, color=:black)
```
The sun is setting at the rate of ``1/20`` radian/min, and appears to be dropping perpendicular to the horizon, as depicted in the figure. How fast is the shadow of a ``25`` meter wall lengthening at the moment when the shadow is ``25`` meters long?
Let the shadow length be labeled ``x``, as it appears on the ``x`` axis above. Then we have by right-angle trigonometry:
```math
\tan(\theta) = \frac{25}{x}
```
of ``x\tan(\theta) = 25``.
As ``t`` evolves, we know ``d\theta/dt`` but what is ``dx/dt``? Using implicit differentiation yields:
```math
\frac{dx}{dt} \cdot \tan(\theta) + x \cdot (\sec^2(\theta)\cdot \frac{d\theta}{dt}) = 0
```
Substituting known values and identifying ``\theta=\pi/4`` when the shadow length, ``x``, is ``25`` gives:
```math
\frac{dx}{dt} \cdot \tan(\pi/4) + 25 \cdot((4/2) \cdot \frac{-1}{20} = 0
```
This can be solved for the unknown: ``dx/dt = 50/20``.
##### Example
A batter hits a ball toward third base at ``75`` ft/sec and runs toward first base at a rate of ``24`` ft/sec. At what rate does the distance between the ball and the batter change when ``2`` seconds have passed?
We will answer this with `SymPy`. First we create some symbols for the movement of the ball towardsthird base, `b(t)`, the runner toward first base, `r(t)`, and the two velocities. We use symbolic functions for the movements, as we will be differentiating them in time:
```julia
@syms b() r() v_b v_r
d = sqrt(b(t)^2 + r(t)^2)
```
The distance formula applies to give ``d``. As the ball and runner are moving in a perpendicular direction, the formula is easy to apply.
We can differentiate `d` in terms of `t` and in process we also find the derivatives of `b` and `r`:
```julia
db, dr = diff(b(t),t), diff(r(t),t) # b(t), r(t) -- symbolic functions
dd = diff(d,t) # d -- not d(t) -- an expression
```
The slight difference in the commands is due to `b` and `r` being symbolic functions, whereas `d` is a symbolic expression. Now we begin substituting. First, from the problem `db` is just the velocity in the ball's direction, or `v_b`. Similarly for `v_r`:
```julia
ddt = subs(dd, db => v_b, dr => v_r)
```
Now, we can substitute in for `b(t)`, as it is `v_b*t`, etc.:
```julia
ddt₁ = subs(ddt, b(t) => v_b * t, r(t) => v_r * t)
```
This finds the rate of change of time for any `t` with symbolic values of the velocities. (And shows how the answer doesn't actually depend on ``t``.) The problem's answer comes from a last substitution:
```julia
ddt₁(t => 2, v_b => 75, v_r => 24)
```
Were this done by "hand," it would be better to work with distance squared to avoid the expansion of complexity from the square root. That is, using implicit differentiation:
```math
\begin{align*}
d^2 &= b^2 + r^2\\
2d\cdot d' &= 2b\cdot b' + 2r\cdot r'\\
d' &= (b\cdot b' + r \cdot r')/d\\
d' &= (tb'\cdot b' + tr' \cdot r')/d\\
d' &= \left((b')^2 + (r')^2\right) \cdot \frac{t}{d}.
\end{align*}
```
##### Example
```julia; hold=true; echo=false; cache=true
###{{{baseball_been_berry_good}}}
## Secant line approaches tangent line...
function baseball_been_berry_good_graph(n)
v0 = 15
x = (t) -> 50t
y = (t) -> v0*t - 5 * t^2
ns = range(.25, stop=3, length=8)
t = ns[n]
ts = range(0, stop=t, length=50)
xs = map(x, ts)
ys = map(y, ts)
degrees = atand(y(t)/(100-x(t)))
degrees = degrees < 0 ? 180 + degrees : degrees
plt = plot(xs, ys, legend=false, size=fig_size, xlim=(0,150), ylim=(0,15))
plot!(plt, [x(t), 100], [y(t), 0.0], color=:orange)
annotate!(plt, [(55, 4,"θ = $(round(Int, degrees)) degrees"),
(x(t), y(t), "($(round(Int, x(t))), $(round(Int, y(t))))")])
end
caption = L"""
The flight of the ball as being tracked by a stationary outfielder. This ball will go over the head of the player. What can the player tell from the quantity $d\theta/dt$?
"""
n = 8
anim = @animate for i=1:n
baseball_been_berry_good_graph(i)
end
imgfile = tempname() * ".gif"
gif(anim, imgfile, fps = 1)
ImageFile(imgfile, caption)
```
A baseball player stands ``100`` meters from home base. A batter hits the
ball directly at the player so that the distance from home plate is
$x(t)$ and the height is $y(t)$.
The player tracks the flight of the ball in terms of the angle
$\theta$ made between the ball and the player. This will satisfy:
```math
\tan(\theta) = \frac{y(t)}{100 - x(t)}.
```
What is the rate of change of $\theta$ with respect to $t$ in terms of that of $x$ and $y$?
We have by the chain rule and quotient rule:
```math
\sec^2(\theta) \theta'(t) = \frac{y'(t) \cdot (100 - x(t)) - y(t) \cdot (-x'(t))}{(100 - x(t))^2}.
```
If we have $x(t) = 50t$ and $y(t)=v_{0y} t - 5 t^2$ when is the rate of change of the angle happening most quickly?
The formula for $\theta'(t)$ is
```math
\theta'(t) = \cos^2(\theta) \cdot \frac{y'(t) \cdot (100 - x(t)) - y(t) \cdot (-x'(t))}{(100 - x(t))^2}.
```
This question requires us to differentiate *again* in $t$. Since we
have fairly explicit function for $x$ and $y$, we will use `SymPy` to
do this.
```julia;
@syms theta()
v0 = 5
x(t) = 50t
y(t) = v0*t - 5 * t^2
eqn = tan(theta(t)) - y(t) / (100 - x(t))
```
```julia;
thetap = diff(theta(t),t)
dtheta = solve(diff(eqn, t), thetap)[1]
```
We could proceed directly by evaluating:
```julia;
d2theta = diff(dtheta, t)(thetap => dtheta)
```
That is not so tractable, however.
It helps to simplify
$\cos^2(\theta(t))$ using basic right-triangle trigonometry. Recall, $\theta$ comes from a right triangle with
height $y(t)$ and length $(100 - x(t))$. The cosine of this angle will
be $100 - x(t)$ divided by the length of the hypotenuse. So we can
substitute:
```julia;
dtheta₁ = dtheta(cos(theta(t))^2 => (100 -x(t))^2/(y(t)^2 + (100-x(t))^2))
```
Plotting reveals some interesting things. For $v_{0y} < 10$ we have graphs that look like:
```julia;
plot(dtheta₁, 0, v0/5)
```
The ball will drop in front of the player, and the change in $d\theta/dt$ is monotonic.
But let's rerun the code with $v_{0y} > 10$:
```julia; hold=true
v0 = 15
x(t) = 50t
y(t) = v0*t - 5 * t^2
eqn = tan(theta(t)) - y(t) / (100 - x(t))
thetap = diff(theta(t),t)
dtheta = solve(diff(eqn, t), thetap)[1]
dtheta₁ = subs(dtheta, cos(theta(t))^2, (100 - x(t))^2/(y(t)^2 + (100 - x(t))^2))
plot(dtheta₁, 0, v0/5)
```
In the second case we have a different shape. The graph is not
monotonic, and before the peak there is an inflection point. Without
thinking too hard, we can see that the greatest change in the angle is
when it is just above the head ($t=2$ has $x(t)=100$).
That these two graphs differ so, means that the player may be able to
read if the ball is going to go over his or her head by paying
attention to the how the ball is being tracked.
##### Example
Hipster pour-over coffee is made with a conical coffee filter. The
cone is actually a [frustum](http://en.wikipedia.org/wiki/Frustum) of
a cone with small diameter, say $r_0$, chopped off. We will parameterize
our cone by a value $h \geq 0$ on the $y$ axis and an angle $\theta$
formed by a side and the $y$ axis. Then the coffee filter is the part
of the cone between some $h_0$ (related $r_0=h_0 \tan(\theta)$) and $h$.
The volume of a cone of height $h$ is $V(h) = \pi/3 h \cdot
R^2$. From the geometry, $R = h\tan(\theta)$. The volume of the
filter then is:
```math
V = V(h) - V(h_0).
```
What is $dV/dh$ in terms of $dR/dh$?
Differentiating implicitly gives:
```math
\frac{dV}{dh} = \frac{\pi}{3} ( R(h)^2 + h \cdot 2 R \frac{dR}{dh}).
```
We see that it depends on $R$ and the change in $R$ with respect to $h$. However, we visualize $h$ - the height - so it is better to re-express. Clearly, $dR/dh = \tan\theta$ and using $R(h) = h \tan(\theta)$ we get:
```math
\frac{dV}{dh} = \pi h^2 \tan^2(\theta).
```
The rate of change goes down as $h$ gets smaller ($h \geq h_0$) and gets bigger for bigger $\theta$.
How do the quantities vary in time?
For an incompressible fluid, by balancing the volume leaving with how
it leaves we will have $dh/dt$ is the ratio of the cross-sectional
area at bottom over that at the height of the fluid $(\pi \cdot (h_0\tan(\theta))^2) /
(\pi \cdot ((h\tan\theta))^2)$ times the outward velocity of the fluid.
That is $dh/dt = (h_0/h)^2 \cdot v$. Which makes sense - larger openings
($h_0$) mean more fluid lost per unit time so the height change
follows, higher levels ($h$) means the change in height is slower, as
the cross-sections have more volume.
By [Torricelli's](http://en.wikipedia.org/wiki/Torricelli's_law) law,
the out velocity follows the law $v = \sqrt{2g(h-h_0)}$. This gives:
```math
\frac{dh}{dt} = \frac{h_0^2}{h^2} \cdot v = \frac{h_0^2}{h^2} \sqrt{2g(h-h_0)}.
```
If $h >> h_0$, then $\sqrt{h-h_0} = \sqrt{h}\sqrt(1 - h_0/h) \approx \sqrt{h}(1 - (1/2)(h_0/h)) \approx \sqrt{h}$. So the rate of change of height in time is like $1/h^{3/2}$.
Now, by the chain rule, we have then the rate of change of volume with respect to time, $dV/dt$, is:
```math
\begin{align*}
\frac{dV}{dt} &=
\frac{dV}{dh} \cdot \frac{dh}{dt}\\
&= \pi h^2 \tan^2(\theta) \cdot \frac{h_0^2}{h^2} \sqrt{2g(h-h_0)} \\
&= \pi \sqrt{2g} \cdot (r_0)^2 \cdot \sqrt{h-h_0} \\
&\approx \pi \sqrt{2g} \cdot r_0^2 \cdot \sqrt{h}.
\end{align*}
```
This rate depends on the square of the size of the
opening ($r_0^2$) and the square root of the height ($h$), but not the
angle of the cone.
## Questions
###### Question
Supply and demand. Suppose demand for product $XYZ$ is $d(x)$ and supply
is $s(x)$. The excess demand is $d(x) - s(x)$. Suppose this is positive. How does this influence
price? Guess the "law" of economics that applies:
```julia; hold=true; echo=false
choices = [
"The rate of change of price will be ``0``",
"The rate of change of price will increase",
"The rate of change of price will be positive and will depend on the rate of change of excess demand."
]
ans = 3
radioq(choices, ans, keep_order=true)
```
(Theoretically, when demand exceeds supply, prices increase.)
###### Question
Which makes more sense from an economic viewpoint?
```julia; hold=true; echo=false
choices = [
"If the rate of change of unemployment is negative, the rate of change of wages will be negative.",
"If the rate of change of unemployment is negative, the rate of change of wages will be positive."
]
ans = 2
radioq(choices, ans, keep_order=true)
```
(Colloquially, "the rate of change of unemployment is negative" means the unemployment rate is going down, so there are fewer workers available to fill new jobs.)
###### Question
In chemistry there is a fundamental relationship between pressure
($P$), temperature ($T)$ and volume ($V$) given by $PV=cT$ where $c$
is a constant. Which of the following would be true with respect to time?
```julia; hold=true; echo=false
choices = [
L"The rate of change of pressure is always increasing by $c$",
"If volume is constant, the rate of change of pressure is proportional to the temperature",
"If volume is constant, the rate of change of pressure is proportional to the rate of change of temperature",
"If pressure is held constant, the rate of change of pressure is proportional to the rate of change of temperature"]
ans = 3
radioq(choices, ans, keep_order=true)
```
###### Question
A pebble is thrown into a lake causing ripples to form expanding
circles. Suppose one of the circles expands at a rate of ``1`` foot per second and
the radius of the circle is ``10`` feet, what is the rate of change of
the area enclosed by the circle?
```julia; hold=true; echo=false
# a = pi*r^2
# da/dt = pi * 2r * drdt
r = 10; drdt = 1
val = pi * 2r * drdt
numericq(val, units=L"feet$^2$/second")
```
###### Question
A pizza maker tosses some dough in the air. The dough is formed in a
circle with radius ``10``. As it rotates, its area increases at a rate of
``1`` inch$^2$ per second. What is the rate of change of the radius?
```julia; hold=true; echo=false
# a = pi*r^2
# da/dt = pi * 2r * drdt
r = 10; dadt = 1
val = dadt /( pi * 2r)
numericq(val, units="inches/second")
```
###### Question
An FBI agent with a powerful spyglass is located in a boat anchored
400 meters offshore. A gangster under surveillance is driving along
the shore. Assume the shoreline is straight and that the gangster is 1
km from the point on the shore nearest to the boat. If the spyglasses
must rotate at a rate of $\pi/4$ radians per minute to track
the gangster, how fast is the gangster moving? (In kilometers per minute.)
[Source.](http://oregonstate.edu/instruct/mth251/cq/Stage9/Practice/ratesProblems.html)
```julia; hold=true; echo=false
## tan(theta) = x/y
## sec^2(theta) dtheta/dt = 1/y dx/dt (y is constant)
## dxdt = y sec^2(theta) dtheta/dt
dthetadt = pi/4
y0 = .4; x0 = 1.0
theta = atan(x0/y0)
val = y0 * sec(theta)^2 * dthetadt
numericq(val, units="kilometers/minute")
```
###### Question
A flood lamp is installed on the ground 200 feet from a vertical
wall. A six foot tall man is walking towards the wall at the rate of
4 feet per second. How fast is the tip of his shadow moving down the
wall when he is 50 feet from the wall?
[Source.](http://oregonstate.edu/instruct/mth251/cq/Stage9/Practice/ratesProblems.html)
(As the question is written the answer should be positive.)
```julia; hold=true; echo=false
## y/200 = 6/x
## dydt = 200 * 6 * -1/x^2 dxdt
x0 = 200 - 50
dxdt = 4
val = 200 * 6 * (1/x0^2) * dxdt
numericq(val, units="feet/second")
```
###### Question
Consider the hyperbola $y = 1/x$ and think of it as a slide. A
particle slides along the hyperbola so that its x-coordinate is
increasing at a rate of $f(x)$ units/sec. If its $y$-coordinate is
decreasing at a constant rate of $1$ unit/sec, what is $f(x)$?
[Source.](http://oregonstate.edu/instruct/mth251/cq/Stage9/Practice/ratesProblems.html)
```julia; hold=true; echo=false
choices = [
"``f(x) = 1/x``",
"``f(x) = x^0``",
"``f(x) = x``",
"``f(x) = x^2``"
]
ans = 4
radioq(choices, ans, keep_order=true)
```
###### Question
A balloon is in the shape of a sphere, fortunately, as this gives
a known formula, $V=4/3 \pi r^3$, for the volume. If the balloon is being filled with a rate of
change of volume per unit time is $2$ and the radius is $3$, what is
rate of change of radius per unit time?
```julia; hold=true; echo=false
r, dVdt = 3, 2
drdt = dVdt / (4 * pi * r^2)
numericq(drdt, units="units per unit time")
```
###### Question
Consider the curve $f(x) = x^2 - \log(x)$. For a given $x$, the tangent line intersects the $y$ axis. Where?
```julia; hold=true; echo=false
choices = [
"``y = 1 - x^2 - \\log(x)``",
"``y = 1 - x^2``",
"``y = 1 - \\log(x)``",
"``y = x(2x - 1/x)``"
]
ans = 1
radioq(choices, ans)
```
If $dx/dt = -1$, what is $dy/dt$?
```julia; hold=true; echo=false
choices = [
"``dy/dt = 2x + 1/x``",
"``dy/dt = 1 - x^2 - \\log(x)``",
"``dy/dt = -2x - 1/x``",
"``dy/dt = 1``"
]
ans=1
radioq(choices, ans)
```

218
CwJ/derivatives/symbolic_derivatives.jmd Normal file
View File

@@ -0,0 +1,218 @@
# Symbolic derivatives
This section uses this add-on package:
```julia
using TermInterface
```
```julia; echo=false
const frontmatter = (
title = "Symbolic derivatives",
description = "Calculus with Julia: Symbolic derivatives",
tags = ["CalculusWithJulia", "derivatives", "symbolic derivatives"],
);
```
----
The ability to breakdown an expression into operations and their
arguments is necessary when trying to apply the differentiation
rules. Such rules are applied from the outside in. Identifying
the proper "outside" function is usually most of the battle when finding derivatives.
In the following example, we provide a sketch of a framework to differentiate expressions by a chosen symbol to illustrate how the outer function drives the task of differentiation.
The `Symbolics` package provides native symbolic manipulation abilities for `Julia`, similar to `SymPy`, though without the dependence on `Python`. The `TermInterface` package, used by `Symbolics`, provides a generic interface for expression manipulation for this package that *also* is implemented for `Julia`'s expressions and symbols.
An expression is an unevaluated portion of code that for our purposes
below contains other expressions, symbols, and numeric literals. They
are held in the `Expr` type. A symbol, such as `:x`, is distinct from
a string (e.g. `"x"`) and is useful to the programmer to distinguish
between the contents a variable points to from the name of the
variable. Symbols are fundamental to metaprogramming in `Julia`. An
expression is a specification of some set of statements to execute. A
numeric literal is just a number.
The three main functions from `TermInterface` we leverage are `istree`, `operation`, and `arguments`. The `operation` function returns the "outside" function of an expression. For example:
```julia
operation(:(sin(x)))
```
We see the `sin` function, referred to by a symbol (`:sin`).
The `:(...)` above *quotes* the argument, and does not evaluate it, hence `x` need not be defined above. (The `:` notation is used to create both symbols and expressions.)
The arguments are the terms that the outside function is called on. For our purposes there may be ``1`` (*unary*), ``2`` (*binary*), or more than ``2`` (*nary*) arguments. (We ignore zero-argument functions.) For example:
```julia
arguments(:(-x)), arguments(:(pi^2)), arguments(:(1 + x + x^2))
```
(The last one may be surprising, but all three arguments are passed to the `+` function.)
Here we define a function to decide the *arity* of an expression based on the number of arguments it is called with:
```julia
function arity(ex)
n = length(arguments(ex))
n == 1 ? Val(:unary) :
n == 2 ? Val(:binary) : Val(:nary)
end
```
Differentiation must distinguish between expressions, variables, and
numbers. Mathematically expressions have an "outer" function, whereas variables and numbers can be directly differentiated. The `istree`
function in `TermInterface` returns `true` when passed an expression,
and `false` when passed a symbol or numeric literal. The latter two
may be distinguished by `isa(..., Symbol)`.
Here we create a function, `D`, that when it encounters an expression it *dispatches* to a specific method of `D` based on the outer operation and arity, otherwise if it encounters a symbol or a numeric literal it does the differentiation:
```julia
function D(ex, var=:x)
if istree(ex)
op, args = operation(ex), arguments(ex)
D(Val(op), arity(ex), args, var)
elseif isa(ex, Symbol) && ex == :x
1
else
0
end
end
```
Now to develop methods for `D` for different "outside" functions and arities.
Addition can be unary (`:(+x)` is a valid quoting, even if it might simplify to the symbol `:x` when evaluated), *binary*, or *nary*. Here we implement the *sum rule*:
```julia
D(::Val{:+}, ::Val{:unary}, args, var) = D(first(args), var)
function D(::Val{:+}, ::Val{:binary}, args, var)
a′, b′ = D.(args, var)
:($a′ + $b′)
end
function D(::Val{:+}, ::Val{:nary}, args, var)
a′s = D.(args, var)
:(+($a′s...))
end
```
The `args` are always held in a container, so the unary method must pull out the first one. The binary case should read as: apply `D` to each of the two arguments, and then create a quoted expression containing the sum of the results. The dollar signs interpolate into the quoting. (The "primes" are unicode notation achieved through `\prime[tab]` and not operations.) The *nary* case does something similar, only uses splatting to produce the sum.
Subtraction must also be implemented in a similar manner, but not for the *nary* case:
```julia
function D(::Val{:-}, ::Val{:unary}, args, var)
a′ = D(first(args), var)
:(-$a′)
end
function D(::Val{:-}, ::Val{:binary}, args, var)
a′, b′ = D.(args, var)
:($a′ - $b′)
end
```
The *product rule* is similar to addition, in that ``3`` cases are considered:
```julia
D(op::Val{:*}, ::Val{:unary}, args, var) = D(first(args), var)
function D(::Val{:*}, ::Val{:binary}, args, var)
a, b = args
a′, b′ = D.(args, var)
:($a′ * $b + $a * $b′)
end
function D(op::Val{:*}, ::Val{:nary}, args, var)
a, bs... = args
b = :(*($(bs...)))
a′ = D(a, var)
b′ = D(b, var)
:($a′ * $b + $a * $b′)
end
```
The *nary* case above just peels off the first factor and then uses the binary product rule.
Division is only a binary operation, so here we have the *quotient rule*:
```julia
function D(::Val{:/}, ::Val{:binary}, args, var)
u,v = args
u′, v′ = D(u, var), D(v, var)
:( ($u′*$v - $u*$v′)/$v^2 )
end
```
Powers are handled a bit differently. The power rule would require checking if the exponent does not contain the variable of differentiation, exponential derivatives would require checking the base does not contain the variable of differentation. Trying to implement both would be tedious, so we use the fact that ``x = \exp(\log(x))`` (for `x` in the domain of `log`, more care is necessary if `x` is negative) to differentiate:
```julia
function D(::Val{:^}, ::Val{:binary}, args, var)
a, b = args
D(:(exp($b*log($a))), var) # a > 0 assumed here
end
```
That leaves the task of defining a rule to differentiate both `exp` and `log`.
We do so with *unary* definitions. In the following we also implement `sin` and `cos` rules:
```julia
function D(::Val{:exp}, ::Val{:unary}, args, var)
a = first(args)
a′ = D(a, var)
:(exp($a) * $a′)
end
function D(::Val{:log}, ::Val{:unary}, args, var)
a = first(args)
a′ = D(a, var)
:(1/$a * $a′)
end
function D(::Val{:sin}, ::Val{:unary}, args, var)
a = first(args)
a′ = D(a, var)
:(cos($a) * $a′)
end
function D(::Val{:cos}, ::Val{:unary}, args, var)
a = first(args)
a′ = D(a, var)
:(-sin($a) * $a′)
end
```
The pattern is similar for each. The `$a′` factor is needed due to the *chain rule*. The above illustrates the simple pattern necessary to add a derivative rule for a function. More could be, but for this example the above will suffice, as now the system is ready to be put to work.
```julia
ex₁ = :(x + 2/x)
D(ex₁, :x)
```
The output does not simplify, so some work is needed to identify `1 - 2/x^2` as the answer.
```julia
ex₂ = :( (x + sin(x))/sin(x))
D(ex₂, :x)
```
Again, simplification is not performed.
Finally, we have a second derivative taken below:
```julia
ex₃ = :(sin(x) - x - x^3/6)
D(D(ex₃, :x), :x)
```
The length of the expression should lead to further appreciation for simplification steps taken when doing such a computation by hand.

1222
CwJ/derivatives/taylor_series_polynomials.jmd Normal file
View File

File diff suppressed because it is too large Load Diff