daviddoji/pbdl-book
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

updated texts for PG code example

Browse Source
This commit is contained in:
NT 2021-03-24 16:38:52 +08:00
parent b3dc5e073a
commit 2a84697faf
1 changed files with 9 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
physgrad-comparison.ipynb
View File

@ -212,14 +212,13 @@
"For gradient descent, the simple gradient based update from equation {eq}`GD-update`\n", "For gradient descent, the simple gradient based update from equation {eq}`GD-update`\n",
"in our setting gives the following update step in $\\mathbf{x}$:\n", "in our setting gives the following update step in $\\mathbf{x}$:\n",
"\n", "\n",
"$\\begin{aligned}\n", "$$\\begin{aligned}\n",
"\\quad\n",
"\\Delta \\mathbf{x} \n", "\\Delta \\mathbf{x} \n",
"&= \n", "&= \n",
"- \\eta ( J_{L} J_{\\mathbf{z}} )^T \\\\\n", "- \\eta ( J_{L} J_{\\mathbf{z}} )^T \\\\\n",
"&=\n", "&=\n",
"- \\eta ( \\frac{\\partial L }{ \\partial \\mathbf{z} } \\frac{\\partial \\mathbf{z} }{ \\partial \\mathbf{x} } )^T\n", "- \\eta ( \\frac{\\partial L }{ \\partial \\mathbf{z} } \\frac{\\partial \\mathbf{z} }{ \\partial \\mathbf{x} } )^T\n",
"\\end{aligned}$\n", "\\end{aligned}$$\n",
"\n", "\n",
"where $\\eta$ denotes the step size parameter .\n", "where $\\eta$ denotes the step size parameter .\n",
"\n", "\n",
@ -324,15 +323,16 @@
"\n", "\n",
"For Newton's method, the update step is given by\n", "For Newton's method, the update step is given by\n",
"\n", "\n",
"$\\begin{aligned}\n", "$$\n",
"\\quad\n", "\\begin{aligned}\n",
"\\Delta \\mathbf{x} &= \n", "\\Delta \\mathbf{x} &= \n",
"- \\eta \\left( \\frac{\\partial^2 L }{ \\partial \\mathbf{x}^2 } \\right)^{-1}\n", "- \\eta \\left( \\frac{\\partial^2 L }{ \\partial \\mathbf{x}^2 } \\right)^{-1}\n",
" \\frac{\\partial L }{ \\partial \\mathbf{x} }\n", " \\frac{\\partial L }{ \\partial \\mathbf{x} }\n",
"\\\\\n", "\\\\\n",
"&=\n", "&=\n",
"- \\eta \\ H_L^{-1} \\ ( J_{L} J_{\\mathbf{z}} )^T\n", "- \\eta \\ H_L^{-1} \\ ( J_{L} J_{\\mathbf{z}} )^T\n",
"\\end{aligned}$\n", "\\end{aligned}\n",
"$$\n",
"\n", "\n",
"Hence, in addition to the same gradient as for GD, we now need to evaluate and invert the Hessian of $\\frac{\\partial^2 L }{ \\partial \\mathbf{x}^2 }$.\n", "Hence, in addition to the same gradient as for GD, we now need to evaluate and invert the Hessian of $\\frac{\\partial^2 L }{ \\partial \\mathbf{x}^2 }$.\n",
"\n", "\n",
@ -444,16 +444,15 @@
"## Physical Gradients\n", "## Physical Gradients\n",
"\n", "\n",
"Now we also use inverse physics, i.e. the inverse of z:\n", "Now we also use inverse physics, i.e. the inverse of z:\n",
"$\\mathbf{z}^{-1}(\\mathbf{x}) = [x_0 \\ x_1^{1/2}]^T$, to compute the _physical gradient_. As a slight look-ahead to the next section, we'll use a Newton's step for $L$, and combine it with the inverse physics function to get an overall update.\n", "$\\mathbf{z}^{-1}(\\mathbf{x}) = [x_0 \\ x_1^{1/2}]^T$, to compute the _physical gradient_. As a slight look-ahead to the next section, we'll use a Newton's step for $L$, and combine it with the inverse physics function to get an overall update. This gives an update step:\n",
"\n", "\n",
"This gives an update step:\n", "$$\\begin{aligned}\n",
"$\\begin{aligned}\n",
"\\Delta \\mathbf{x} &= \n", "\\Delta \\mathbf{x} &= \n",
"\\mathbf{z}^{-1} \\left( \\mathbf{z}(\\mathbf{x}) - \\eta\n", "\\mathbf{z}^{-1} \\left( \\mathbf{z}(\\mathbf{x}) - \\eta\n",
" \\left( \\frac{\\partial^2 L }{ \\partial \\mathbf{z}^2 } \\right)^{-1}\n", " \\left( \\frac{\\partial^2 L }{ \\partial \\mathbf{z}^2 } \\right)^{-1}\n",
" \\frac{\\partial L }{ \\partial \\mathbf{z} }\n", " \\frac{\\partial L }{ \\partial \\mathbf{z} }\n",
"\\right) - \\mathbf{x}\n", "\\right) - \\mathbf{x}\n",
"\\end{aligned}$\n", "\\end{aligned}$$\n",
"\n", "\n",
"Below, we define our inverse function `fun_z_inv_analytic` (we'll come to a variant below), and then evaluate an optimization with the physical gradient for ten steps:\n" "Below, we define our inverse function `fun_z_inv_analytic` (we'll come to a variant below), and then evaluate an optimization with the physical gradient for ten steps:\n"
] ]