updated HIG code example
This commit is contained in:
NT committed
1 parent
0e3cd32658
commit
aa4f67eec7
1 file changed
+367
-204
+367
-204
@@ -2,70 +2,116 @@
|
|||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
"source": [
|
"id": "2tT-NooLyyGW"
|
||||||
"# Comparing Half-Inverse Gradients to Other Methods\n",
|
|
||||||
"\n",
|
|
||||||
"In this notebook, we demonstrate three different ways to optimize neural networks with physical loss functions. We'll compare: \n",
|
|
||||||
"\n",
|
|
||||||
"1. Adam; a standard network optimizer\n",
|
|
||||||
"2. Physical Gradients; an algorithm fully inverting the physics\n",
|
|
||||||
"3. Half-Inverse Gradients; an algorithm locally inverting physics and network.\n",
|
|
||||||
"\n",
|
|
||||||
"The learning task is to find the control function steering the dynamics of a coupled oscillator system. More concretely, we consider a set of different phyiscal inputs ($x_i$). Using a corresponding control function ($u_i$), we can influence the time evolution ($T$) and receive an output state ($y_i$).\n",
|
|
||||||
"\n",
|
|
||||||
"$y_i=T(x_i;u_i)$\n",
|
|
||||||
"\n",
|
|
||||||
"If we want to evolve a given initial state($x_i$) into a given target state ($\\hat{y_i}$), we receive an inverse problem for the control function $(u_i)$. The quality of between desired target state ($\\hat{y}_i$) und received target state ($y_i$) is measured by a loss function ($L$).\n",
|
|
||||||
"\n",
|
|
||||||
"$\\min_{u_i}\\; L(\\hat{y}_i,T(x_i;u_i))$\n",
|
|
||||||
"\n",
|
|
||||||
"If we use a neural network ($N$ parameterized by $\\theta$) to learn the control function over a set of input/output pairs ($x_i,\\hat{y}_i$), we transform the above physics optimization task into a learning problem.\n",
|
|
||||||
"\n",
|
|
||||||
"$\\min_{\\theta}\\; \\sum_i L(\\hat{y}_i,T(x_i;N(x_i,y_i,\\theta)))$\n",
|
|
||||||
"\n",
|
|
||||||
"We begin by setting up the physics solver ($T$), the network ($N$), and the loss ($L$)."
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": 77,
|
|
||||||
"metadata": {},
|
|
||||||
"outputs": [],
|
|
||||||
"source": [
|
"source": [
|
||||||
"import numpy as np\n",
|
"# Coupled Oscillators with Half-Inverse Gradients\n",
|
||||||
"import tensorflow as tf\n",
|
|
||||||
"import time\n",
|
|
||||||
"import os\n",
|
|
||||||
"\n",
|
"\n",
|
||||||
"os.environ[\"CUDA_VISIBLE_DEVICES\"]='' # set GPUs\n",
|
"In this notebook, we'll turn to a practical example comparing the half-inverse gradients (HIGs) to other methods for training neural networks with physical loss functions. Specifically, we'll compare:\n",
|
||||||
"#tf.config.run_functions_eagerly(True) # deactivate Tensorflow Tracing"
|
"\n",
|
||||||
|
"1. Adam: as a standard gradient-descent (GD) based network optimizer,\n",
|
||||||
|
"2. Scale-Invariant Physics: the previously described algorithm that fully inverts the physics,\n",
|
||||||
|
"3. Half-Inverse Gradients: which locally and jointly inverting physics and network.\n",
|
||||||
|
"\n",
|
||||||
|
"\n",
|
||||||
|
"\n",
|
||||||
|
"## Inverse problem setup\n",
|
||||||
|
"\n",
|
||||||
|
"The learning task is to find the control function steering the dynamics of a coupled oscillator system. \n",
|
||||||
|
"This is a classical problem in physics, and a good case to evaluate the HIGs due to it's smaller size. We're using two mass points, and thus we'll only have four degrees of freedom for position and velocity of both points (compared to, e.g., the $32\\times32\\times2$ unknowns we'd get even for \"only\" a small fluid simulation with 32 cells along x and y). \n",
|
||||||
|
"\n",
|
||||||
|
"Nonetheless, the oscillators are a highly-non trivial case: we aim for applying a control such that the initial state is reached again after a chosen time interval. We'll use 24 steps of a fourth-order Runge-Kutta scheme, and hence the NN has to learn how to best \"nudge\" the two mass points over the course of all time steps, so that they end up at the desired position with the right velocity at the right time.\n",
|
||||||
|
"\n",
|
||||||
|
"A system of $N$ coupled oscillators is described by the following Hamiltonian:\n",
|
||||||
|
"\n",
|
||||||
|
"$$\n",
|
||||||
|
" \\mathcal{H}(x_i,p_i,t)=\\sum_i \\bigg( \\frac{x_i^2}{2}+ \\frac{p_i^2}{2} + \\alpha \\cdot (x_i-x_{i+1})^4+u(t) \\cdot x_i \\cdot c_i\\bigg),\n",
|
||||||
|
"$$\n",
|
||||||
|
"\n",
|
||||||
|
"which provides the basis for the RK4 time integration below.\n",
|
||||||
|
"\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
|
||||||
"source": [
|
"source": [
|
||||||
"## Physics Model\n",
|
|
||||||
"\n",
|
"\n",
|
||||||
"Differential equation: Coupled Linear Oscillators with Control term\n",
|
"## Problem statement\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Solver: Runge-Kutta-4 \n",
|
"More concretely, we consider a set of different phyiscal inputs ($x_i$). Using a corresponding control function ($u_i$), we can influence the time evolution of our physical system $\\mathcal P$ and receive an output state ($y_i$).\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Nx: Number of oscillators,\n",
|
"$$y_i=\\mathcal P(x_i;u_i)$$\n",
|
||||||
"Nt: Number of time evolution steps,\n",
|
"\n",
|
||||||
"dt: Length of one time step"
|
"If we want to evolve a given initial state($x_i$) into a given target state ($\\hat{y_i}$), we receive an inverse problem for the control function $(u_i)$. The quality of between desired target state ($\\hat{y}_i$) und received target state ($y_i$) is measured by a loss function ($L$).\n",
|
||||||
|
"\n",
|
||||||
|
"$$\\text{arg min}_{u_i}\\; \\mathcal P(\\hat{y}_i,T(x_i;u_i))$$\n",
|
||||||
|
"\n",
|
||||||
|
"If we use a neural network ($f$ parameterized by $\\theta$) to learn the control function over a set of input/output pairs ($x_i,\\hat{y}_i$), we transform the above physics optimization task into a learning problem.\n",
|
||||||
|
"\n",
|
||||||
|
"$$\\text{arg min}_{\\theta}\\; \\sum_i L\\big(\\hat{y}_i, \\mathcal P(x_i;f(x_i,y_i; \\theta)) \\big)$$"
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"id": "5NOyhZ-MOnoZ"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"source": [
|
||||||
|
"Before we begin by setting up the physics solver $\\mathcal P$, we import the necessary libraries (this example uses TensorFlow), and define the main global variable `MODE`, that switches between _Adam_ (`'GD'`), _Scale-invariant physics_ (`'SIP'`), and _Half-inverse gradients_ (`'HIG'`)."
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"id": "We6eG_ZWOspS"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 95,
|
||||||
|
"metadata": {
|
||||||
|
"id": "A0cyHl7tyyGg"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"import numpy as np\n",
|
||||||
|
"import tensorflow as tf\n",
|
||||||
|
"import time, os\n",
|
||||||
|
"\n",
|
||||||
|
"os.environ[\"CUDA_VISIBLE_DEVICES\"]='' # set GPUs\n",
|
||||||
|
"#tf.config.run_functions_eagerly(True) # deactivate Tensorflow Tracing\n",
|
||||||
|
"\n",
|
||||||
|
"# main switch for the three methods:\n",
|
||||||
|
"MODE = 'HIG' # HIG | SIP | GD"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "yCN4F_AjyyGk"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"## Coupled linear oscillator simulation\n",
|
||||||
|
"\n",
|
||||||
|
"For the physics simulation, we'll solve a differential equation for a system of coupled linear oscillators with a control term. The time integration, a fourth-order Runge-Kutta scheme is used.\n",
|
||||||
|
"\n",
|
||||||
|
"Below, we're first defining a few global constants:\n",
|
||||||
|
"`Nx`: the number of oscillators,\n",
|
||||||
|
"`Nt`: the number of time evolution steps, and\n",
|
||||||
|
"`DT`: the length of one time step. \n",
|
||||||
|
"\n",
|
||||||
|
"We'll then define a helper function to set up a Laplacian stencil, the `coupled_oscillators_batch()` function which computes the simulation for a whole mini batch of values, and finally the `solver()` function, which runs the desired number of time steps with a given control signal."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 78,
|
"execution_count": 96,
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "hevmSaVvyyGl"
|
||||||
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"Nx = 2\n",
|
"Nx = 2\n",
|
||||||
"Nt = 24\n",
|
"Nt = 24\n",
|
||||||
"dt = 0.5\n",
|
"DT = 0.5\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def build_laplace(n,boundary='0'):\n",
|
"def build_laplace(n,boundary='0'):\n",
|
||||||
" if n==1:\n",
|
" if n==1:\n",
|
||||||
@@ -83,14 +129,13 @@
|
|||||||
"\n",
|
"\n",
|
||||||
"@tf.function\n",
|
"@tf.function\n",
|
||||||
"def coupled_oscillators_batch( x, control):\n",
|
"def coupled_oscillators_batch( x, control):\n",
|
||||||
" print('coup harm osc tracing')\n",
|
|
||||||
" '''\n",
|
" '''\n",
|
||||||
" ODE of type: x' = f(x)\n",
|
" ODE of type: x' = f(x)\n",
|
||||||
" :param x_in: shape: (batch, 2 * number of osc)\n",
|
" :param x_in: position and velocities, shape: (batch, 2 * number of osc) order second index: x_i , v_i\n",
|
||||||
" order second index: x_i , v_i\n",
|
" :param control: control function, shape: (batch,)\n",
|
||||||
" :param control: shape:(batch,)\n",
|
|
||||||
" :return:\n",
|
" :return:\n",
|
||||||
" '''\n",
|
" '''\n",
|
||||||
|
" #print('coupled_oscillators_batch')\n",
|
||||||
" n_osc = x.shape[1]//2\n",
|
" n_osc = x.shape[1]//2\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # natural time evo\n",
|
" # natural time evo\n",
|
||||||
@@ -139,21 +184,27 @@
|
|||||||
"def solver(x0, control):\n",
|
"def solver(x0, control):\n",
|
||||||
" x = x0\n",
|
" x = x0\n",
|
||||||
" for i in range(Nt):\n",
|
" for i in range(Nt):\n",
|
||||||
" x = runge_kutta_4_batch(x, dt, control[:,i], coupled_oscillators_batch)\n",
|
" x = runge_kutta_4_batch(x, DT, control[:,i], coupled_oscillators_batch)\n",
|
||||||
" return x\n"
|
" return x\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "VtY4-wbXyyGn"
|
||||||
|
},
|
||||||
"source": [
|
"source": [
|
||||||
"## Network Model"
|
"## Training setup\n",
|
||||||
|
"\n",
|
||||||
|
"The neural network itself is quite simple: it consists of four dense layers (the intermediate ones with 20 neurons each), and `tanh` activation functions."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 79,
|
"execution_count": 97,
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "et1j8lYRyyGp"
|
||||||
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"act = tf.keras.activations.tanh\n",
|
"act = tf.keras.activations.tanh\n",
|
||||||
@@ -168,15 +219,19 @@
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "zilE67gYyyGq"
|
||||||
|
},
|
||||||
"source": [
|
"source": [
|
||||||
"## Loss Function"
|
"As loss function, we'll use an $L^2$ loss:"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 80,
|
"execution_count": 98,
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "km83dkWgyyGq"
|
||||||
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"@tf.function\n",
|
"@tf.function\n",
|
||||||
@@ -189,95 +244,104 @@
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "ZyS31DHdyyGr"
|
||||||
|
},
|
||||||
"source": [
|
"source": [
|
||||||
"## Data set\n",
|
"And as data set for training we simply create 4k of random position values which the oscillators start with (`X_TRAIN`), and which they should return to at the end of the simulation (`Y_TRAIN`). As they should return to their initial states, we have `X_TRAIN=Y_TRAIN`.\n",
|
||||||
"\n"
|
"\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 81,
|
"execution_count": 99,
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "_zz7VSIpyyGs"
|
||||||
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"N = 2**12\n",
|
"N = 2**12\n",
|
||||||
"x_train = np.random.rand(N, 2 * Nx).astype(np.float32)\n",
|
"X_TRAIN = np.random.rand(N, 2 * Nx).astype(np.float32)\n",
|
||||||
"y_train = x_train"
|
"Y_TRAIN = X_TRAIN # the target states are identical"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "1KChFddqyyGs"
|
||||||
|
},
|
||||||
"source": [
|
"source": [
|
||||||
"## Optimization\n",
|
"## Training\n",
|
||||||
"For the optimization procedure, we need to set up some parameters. \n",
|
"For the optimization procedure of the neural network training, we need to set up some global parameters. The next cell initializes some suitable values tailored to each of the three methods. These were determined heuristically to work best for each. If we try to use the same settings for all, this would inevitably make the comparison unfair for some of them.\n",
|
||||||
"1. Adam: The truncation parameter has no meaning here.\n",
|
|
||||||
"2. PG: The specified optimizer is the one used for network optimization. The physics inversion is done via Gauss-Newton and corresponds to an exact inversion since the physical optimization landscape is quadratic. For the Jacobian inversion in Gauss-Newton, we can specify a truncation parameter.\n",
|
|
||||||
"3. HIG: To receive the HIG algorithm, the optimizer has to be set to SGD. For the Jacobian half-inversion, we can specify a truncation parameter. Optimal batch sizes are typically lower. Learning rates can usually be chosen around 1.\n",
|
|
||||||
"\n",
|
"\n",
|
||||||
"Specify the maximal simulation time in seconds via max_number."
|
"1. _Adam_: This is the most widely used NN optimizer, and we're using it here as a representative of the GD family. Note that the truncation parameter has no meaning for Adam.\n",
|
||||||
|
"\n",
|
||||||
|
"2. _SIP_: The specified optimizer is the one used for network optimization. The physics inversion is done via Gauss-Newton and corresponds to an exact inversion since the physical optimization landscape is quadratic. For the Jacobian inversion in Gauss-Newton, we can specify a truncation parameter.\n",
|
||||||
|
"\n",
|
||||||
|
"3. _HIG_: To obtain the HIG algorithm, the optimizer has to be set to SGD. For the Jacobian half-inversion, we can specify a truncation parameter. Optimal batch sizes are typically lower than for the other two, and a learning rate of 1 typically works very well.\n",
|
||||||
|
"\n",
|
||||||
|
"The maximal training time in seconds is set via `MAX_TIME` below."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 82,
|
"execution_count": 100,
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
"outputs": [],
|
"colab": {
|
||||||
"source": [
|
"base_uri": "https://localhost:8080/"
|
||||||
"# Adam Training\n",
|
|
||||||
"if 0:\n",
|
|
||||||
" mode = 'GD' \n",
|
|
||||||
" optimizer = tf.keras.optimizers.Adam\n",
|
|
||||||
" batch_size = 256\n",
|
|
||||||
" learning_rate = 0.001\n",
|
|
||||||
" max_number = 30#*60\n",
|
|
||||||
" trunc = 10**-10 "
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
{
|
"id": "PYjx7Sk8yyGz",
|
||||||
"cell_type": "code",
|
"outputId": "9ee462ad-499c-4f7c-d3a3-0fb11ab36abe"
|
||||||
"execution_count": 83,
|
|
||||||
"metadata": {},
|
|
||||||
"outputs": [],
|
|
||||||
"source": [
|
|
||||||
"# PG training\n",
|
|
||||||
"if 0:\n",
|
|
||||||
" mode = 'PG' \n",
|
|
||||||
" optimizer = tf.keras.optimizers.Adam\n",
|
|
||||||
" batch_size = 256\n",
|
|
||||||
" learning_rate = 0.001\n",
|
|
||||||
" max_number = 30#*60\n",
|
|
||||||
" trunc = 10**-10"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
|
"outputs": [
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"output_type": "stream",
|
||||||
"execution_count": 84,
|
"name": "stdout",
|
||||||
"metadata": {},
|
"text": [
|
||||||
"outputs": [],
|
"Running variant: HIG\n"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
"source": [
|
"source": [
|
||||||
"# HIG training\n",
|
"if MODE=='HIG': # HIG training\n",
|
||||||
"if 1:\n",
|
" OPTIMIZER = tf.keras.optimizers.SGD\n",
|
||||||
" mode = 'HIG' # GD, PG, HIG\n",
|
" BATCH_SIZE = 32 # larger batches make HIGs unnecessariy slow...\n",
|
||||||
" optimizer = tf.keras.optimizers.SGD\n",
|
" LR = 1.0\n",
|
||||||
" batch_size = 32\n",
|
" TRUNC = 10**-10\n",
|
||||||
" learning_rate = 1.0\n",
|
"\n",
|
||||||
" max_number = 30#*60\n",
|
"elif MODE=='SIP': # SIP training\n",
|
||||||
" trunc = 10**-10"
|
" OPTIMIZER = tf.keras.optimizers.Adam\n",
|
||||||
|
" BATCH_SIZE = 256\n",
|
||||||
|
" LR = 0.001 # for the internal step with Adam\n",
|
||||||
|
" TRUNC = 0 # not used\n",
|
||||||
|
"\n",
|
||||||
|
"else: # Adam Training (as GD representative)\n",
|
||||||
|
" MODE = 'GD' \n",
|
||||||
|
" OPTIMIZER = tf.keras.optimizers.Adam\n",
|
||||||
|
" BATCH_SIZE = 256\n",
|
||||||
|
" LR = 0.001\n",
|
||||||
|
" TRUNC = 0 # not used\n",
|
||||||
|
"\n",
|
||||||
|
"# global parameters for all three methods\n",
|
||||||
|
"MAX_TIME = 100 # [s]\n",
|
||||||
|
"print(\"Running variant: \"+MODE)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "4VFJ4pmEyyG0"
|
||||||
|
},
|
||||||
"source": [
|
"source": [
|
||||||
"Function to construct the half-inverse of matrix for HIGs"
|
"The next function, `HIG_pinv()`, is a crucial one: it constructs the half-inverse of a given matrix for HIGs. It computes an SVD, takes the square-root of the singular values, and then re-assembles the matrix. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 85,
|
"execution_count": 101,
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "c3B4qtUPyyG1"
|
||||||
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from tensorflow.python.framework import ops\n",
|
"from tensorflow.python.framework import ops\n",
|
||||||
@@ -288,9 +352,11 @@
|
|||||||
"from tensorflow.python.util.tf_export import tf_export\n",
|
"from tensorflow.python.util.tf_export import tf_export\n",
|
||||||
"from tensorflow.python.ops.linalg.linalg_impl import _maybe_validate_matrix,svd\n",
|
"from tensorflow.python.ops.linalg.linalg_impl import _maybe_validate_matrix,svd\n",
|
||||||
"\n",
|
"\n",
|
||||||
"@tf_export('linalg.gpinv')\n",
|
"# partial inversion of the Jacobian via SVD:\n",
|
||||||
|
"# this function is adopted from tensorflow's SVD function, and published here under its license: Apache-2.0 License , https://github.com/tensorflow/tensorflow/blob/master/LICENSE\n",
|
||||||
|
"@tf_export('linalg.HIG_pinv')\n",
|
||||||
"@dispatch.add_dispatch_support\n",
|
"@dispatch.add_dispatch_support\n",
|
||||||
"def gpinv(a, rcond=None,beta=0.5, validate_args=False, name=None):\n",
|
"def HIG_pinv(a, rcond=None,beta=0.5, validate_args=False, name=None):\n",
|
||||||
"\n",
|
"\n",
|
||||||
" with ops.name_scope(name or 'pinv'):\n",
|
" with ops.name_scope(name or 'pinv'):\n",
|
||||||
" a = ops.convert_to_tensor(a, name='a')\n",
|
" a = ops.convert_to_tensor(a, name='a')\n",
|
||||||
@@ -321,25 +387,14 @@
|
|||||||
"\n",
|
"\n",
|
||||||
" rcond = ops.convert_to_tensor(rcond, dtype=dtype, name='rcond')\n",
|
" rcond = ops.convert_to_tensor(rcond, dtype=dtype, name='rcond')\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # Calculate pseudo inverse via SVD.\n",
|
" [ singular_values, left_singular_vectors, right_singular_vectors, ] = svd(\n",
|
||||||
" # Note: if a is Hermitian then u == v. (We might observe additional\n",
|
|
||||||
" # performance by explicitly setting `v = u` in such cases.)\n",
|
|
||||||
" [\n",
|
|
||||||
" singular_values, # Sigma\n",
|
|
||||||
" left_singular_vectors, # U\n",
|
|
||||||
" right_singular_vectors, # V\n",
|
|
||||||
" ] = svd(\n",
|
|
||||||
" a, full_matrices=False, compute_uv=True)\n",
|
" a, full_matrices=False, compute_uv=True)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # Saturate small singular values to inf. This has the effect of make\n",
|
|
||||||
" # `1. / s = 0.` while not resulting in `NaN` gradients.\n",
|
|
||||||
" cutoff = rcond * math_ops.reduce_max(singular_values, axis=-1)\n",
|
" cutoff = rcond * math_ops.reduce_max(singular_values, axis=-1)\n",
|
||||||
" singular_values = array_ops.where_v2(\n",
|
" singular_values = array_ops.where_v2(\n",
|
||||||
" singular_values > array_ops.expand_dims_v2(cutoff, -1), singular_values**beta,\n",
|
" singular_values > array_ops.expand_dims_v2(cutoff, -1), singular_values**beta,\n",
|
||||||
" np.array(np.inf, dtype))\n",
|
" np.array(np.inf, dtype))\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # By the definition of the SVD, `a == u @ s @ v^H`, and the pseudo-inverse\n",
|
|
||||||
" # is defined as `pinv(a) == v @ inv(s) @ u^H`.\n",
|
|
||||||
" a_pinv = math_ops.matmul(\n",
|
" a_pinv = math_ops.matmul(\n",
|
||||||
" right_singular_vectors / array_ops.expand_dims_v2(singular_values, -2),\n",
|
" right_singular_vectors / array_ops.expand_dims_v2(singular_values, -2),\n",
|
||||||
" left_singular_vectors,\n",
|
" left_singular_vectors,\n",
|
||||||
@@ -353,18 +408,26 @@
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "Bj90LjpyyyG2"
|
||||||
|
},
|
||||||
"source": [
|
"source": [
|
||||||
"We define a Python class to organize the optimization algorithm. Using physics solver, network model, loss function and a data set. We compute loss values after each epoch. Each epoch consists of several mini batch updates. Depending on the optimization method, the mini batch update differs:\n",
|
"Now we have all pieces in place to run the training. The next cell defines a Python class to organize the neural network optimization. It receives the physics solver, network model, loss function and a data set, and runs as many epochs as possible within the given time limit `MAX_TIME`.\n",
|
||||||
"1. Adam: Compute loss gradient, then apply the Adam\n",
|
"\n",
|
||||||
"2. PG: Compute loss gradient und physics Jacobian, invert them data-point-wise, define a modified loss to compute network updates\n",
|
"Depending on the chosen optimization method, the mini batch updates differ:\n",
|
||||||
"3. HIG: Compute loss gradient and network-physics Jacobian, then compute half-inversion"
|
"1. Adam: Compute loss gradient, then apply the Adam update.\n",
|
||||||
|
"2. PG: Compute loss gradient und physics Jacobian, invert them data-point-wise, and compute network updates via the proxy loss and Adam.\n",
|
||||||
|
"3. HIG: Compute loss gradient and network-physics Jacobian, then jointly compute the half-inversion, and update the network parameters with the resulting step.\n",
|
||||||
|
"\n",
|
||||||
|
"The `mini_batch_update()` method of the optimizer class realizes these three variants."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 86,
|
"execution_count": 102,
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "peqXxMNpyyG5"
|
||||||
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"class Optimization():\n",
|
"class Optimization():\n",
|
||||||
@@ -377,9 +440,9 @@
|
|||||||
" self.y_dim = y_train.shape[1]\n",
|
" self.y_dim = y_train.shape[1]\n",
|
||||||
" self.weight_shapes = [weight_tensor.shape for weight_tensor in self.model.trainable_weights]\n",
|
" self.weight_shapes = [weight_tensor.shape for weight_tensor in self.model.trainable_weights]\n",
|
||||||
"\n",
|
"\n",
|
||||||
" def set_params(self,batch_size,learning_rate,optimizer,max_number,mode,trunc):\n",
|
" def set_params(self,batch_size,learning_rate,optimizer,max_time,mode,trunc):\n",
|
||||||
" self.number_of_batches = N // batch_size\n",
|
" self.number_of_batches = N // batch_size\n",
|
||||||
" self.max_number = max_number\n",
|
" self.max_time = max_time\n",
|
||||||
" self.batch_size = batch_size\n",
|
" self.batch_size = batch_size\n",
|
||||||
" self.learning_rate = learning_rate\n",
|
" self.learning_rate = learning_rate\n",
|
||||||
" self.optimizer = optimizer(learning_rate)\n",
|
" self.optimizer = optimizer(learning_rate)\n",
|
||||||
@@ -406,7 +469,7 @@
|
|||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"\n",
|
||||||
" @tf.function\n",
|
" @tf.function\n",
|
||||||
" def pg_get_physics_derivatives(self,x_batch, y_batch): #physical grads\n",
|
" def pg_get_physics_derivatives(self,x_batch, y_batch): # physics gradient for SIP\n",
|
||||||
"\n",
|
"\n",
|
||||||
" with tf.GradientTape(persistent=True) as tape:\n",
|
" with tf.GradientTape(persistent=True) as tape:\n",
|
||||||
" control_batch = self.model(y_batch)\n",
|
" control_batch = self.model(y_batch)\n",
|
||||||
@@ -456,7 +519,7 @@
|
|||||||
" grad = self.gd_get_derivatives(x_batch, y_batch)\n",
|
" grad = self.gd_get_derivatives(x_batch, y_batch)\n",
|
||||||
" self.optimizer.apply_gradients(zip(grad, self.model.trainable_weights))\n",
|
" self.optimizer.apply_gradients(zip(grad, self.model.trainable_weights))\n",
|
||||||
"\n",
|
"\n",
|
||||||
" elif self.mode==\"PG\":\n",
|
" elif self.mode==\"SIP\":\n",
|
||||||
" jacy,grad,control_batch = self.pg_get_physics_derivatives(x_batch, y_batch)\n",
|
" jacy,grad,control_batch = self.pg_get_physics_derivatives(x_batch, y_batch)\n",
|
||||||
" grad_e = tf.expand_dims(grad,-1)\n",
|
" grad_e = tf.expand_dims(grad,-1)\n",
|
||||||
" pinv = tf.linalg.pinv(jacy,rcond=10**-5)\n",
|
" pinv = tf.linalg.pinv(jacy,rcond=10**-5)\n",
|
||||||
@@ -470,7 +533,7 @@
|
|||||||
" flat_jacy_list = [tf.reshape(jac, (self.batch_size * self.y_dim, -1)) for jac in jacy]\n",
|
" flat_jacy_list = [tf.reshape(jac, (self.batch_size * self.y_dim, -1)) for jac in jacy]\n",
|
||||||
" flat_jacy = tf.concat(flat_jacy_list, axis=1)\n",
|
" flat_jacy = tf.concat(flat_jacy_list, axis=1)\n",
|
||||||
" flat_grad = tf.reshape(grad, (-1,))\n",
|
" flat_grad = tf.reshape(grad, (-1,))\n",
|
||||||
" inv = gpinv(flat_jacy, rcond=self.trunc)\n",
|
" inv = HIG_pinv(flat_jacy, rcond=self.trunc)\n",
|
||||||
" processed_derivatives = tf.tensordot(inv, flat_grad, axes=(1, 0))\n",
|
" processed_derivatives = tf.tensordot(inv, flat_grad, axes=(1, 0))\n",
|
||||||
" #processed_derivatives = self.linear_solve(flat_jacy, flat_grad)\n",
|
" #processed_derivatives = self.linear_solve(flat_jacy, flat_grad)\n",
|
||||||
" update_list = []\n",
|
" update_list = []\n",
|
||||||
@@ -486,18 +549,17 @@
|
|||||||
"\n",
|
"\n",
|
||||||
" def epoch_update(self):\n",
|
" def epoch_update(self):\n",
|
||||||
" for batch_index in range(self.number_of_batches):\n",
|
" for batch_index in range(self.number_of_batches):\n",
|
||||||
" position = batch_index * batch_size\n",
|
" position = batch_index * self.batch_size\n",
|
||||||
" x_batch = self.x_train[position:position + batch_size]\n",
|
" x_batch = self.x_train[position:position + self.batch_size]\n",
|
||||||
" y_batch = self.y_train[position:position + batch_size]\n",
|
" y_batch = self.y_train[position:position + self.batch_size]\n",
|
||||||
" self.mini_batch_update(x_batch, y_batch)\n",
|
" self.mini_batch_update(x_batch, y_batch)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" def eval(self,epoch,wc_time,ep_dur):\n",
|
" def eval(self,epoch,wc_time,ep_dur):\n",
|
||||||
"\n",
|
|
||||||
" print('Epoch: ', epoch,' WallClockTime: ',wc_time,' EpochDuration: ',ep_dur )\n",
|
|
||||||
"\n",
|
|
||||||
" train_loss = self.computation(self.x_train,self.y_train)\n",
|
" train_loss = self.computation(self.x_train,self.y_train)\n",
|
||||||
" train_loss_per_dp = train_loss / N\n",
|
" train_loss_per_dp = train_loss / N\n",
|
||||||
" print('TrainLoss:', train_loss_per_dp)\n",
|
" if epoch<5 or epoch%20==0: print('Epoch: ', epoch,', wall clock time: ',wc_time,', loss: ', float(train_loss_per_dp) )\n",
|
||||||
|
" #print('TrainLoss:', train_loss_per_dp)\n",
|
||||||
|
" #print('Epoch: ', epoch,' WallClockTime: ',wc_time,' EpochDuration: ',ep_dur )\n",
|
||||||
" return train_loss_per_dp\n",
|
" return train_loss_per_dp\n",
|
||||||
"\n",
|
"\n",
|
||||||
" def start_training(self):\n",
|
" def start_training(self):\n",
|
||||||
@@ -509,7 +571,7 @@
|
|||||||
" epoch=0\n",
|
" epoch=0\n",
|
||||||
" wc_time = 0\n",
|
" wc_time = 0\n",
|
||||||
"\n",
|
"\n",
|
||||||
" while wc_time<max_number:\n",
|
" while wc_time<self.max_time:\n",
|
||||||
" \n",
|
" \n",
|
||||||
" duration = time.time()\n",
|
" duration = time.time()\n",
|
||||||
" self.epoch_update()\n",
|
" self.epoch_update()\n",
|
||||||
@@ -530,104 +592,200 @@
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "3hCSV9KcyyG7"
|
||||||
|
},
|
||||||
"source": [
|
"source": [
|
||||||
"Start the simulation."
|
"All that's left to do is to start the training with the chosen global parameters, and collect the results in `time_list`, and `loss_list`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 87,
|
"execution_count": 103,
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"colab": {
|
||||||
|
"base_uri": "https://localhost:8080/"
|
||||||
|
},
|
||||||
|
"id": "6BazTjB5yyG8",
|
||||||
|
"outputId": "f4cf85e6-e7d3-4ed1-8c67-b442e1653bfe"
|
||||||
|
},
|
||||||
"outputs": [
|
"outputs": [
|
||||||
{
|
{
|
||||||
"name": "stdout",
|
|
||||||
"output_type": "stream",
|
"output_type": "stream",
|
||||||
|
"name": "stdout",
|
||||||
"text": [
|
"text": [
|
||||||
"Epoch: 0 WallClockTime: 0 EpochDuration: 0\n",
|
"Epoch: 0 , wall clock time: 0 , loss: 1.4401766061782837\n",
|
||||||
"coup harm osc tracing\n",
|
"Epoch: 1 , wall clock time: 28.06755018234253 , loss: 5.44398972124327e-05\n",
|
||||||
"TrainLoss: tf.Tensor(1.0081012, shape=(), dtype=float32)\n",
|
"Epoch: 2 , wall clock time: 31.38792371749878 , loss: 1.064436037268024e-05\n",
|
||||||
"coup harm osc tracing\n",
|
"Epoch: 3 , wall clock time: 34.690271854400635 , loss: 3.163525434501935e-06\n",
|
||||||
"Epoch: 1 WallClockTime: 27.670811891555786 EpochDuration: 27.670811891555786\n",
|
"Epoch: 4 , wall clock time: 37.9914448261261 , loss: 1.1857609933940694e-06\n"
|
||||||
"TrainLoss: tf.Tensor(6.0117705e-05, shape=(), dtype=float32)\n",
|
|
||||||
"Epoch: 2 WallClockTime: 30.26737689971924 EpochDuration: 2.596565008163452\n",
|
|
||||||
"TrainLoss: tf.Tensor(1.1014066e-05, shape=(), dtype=float32)\n"
|
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"source": [
|
"source": [
|
||||||
"opt = Optimization(model,solver,loss_function,x_train,y_train)\n",
|
"opt = Optimization(model, solver, loss_function, X_TRAIN, Y_TRAIN)\n",
|
||||||
"opt.set_params(batch_size,learning_rate,optimizer,max_number,mode,trunc)\n",
|
"opt.set_params(BATCH_SIZE, LR, OPTIMIZER, MAX_TIME, MODE, TRUNC)\n",
|
||||||
"time_list, loss_list = opt.start_training()"
|
"time_list, loss_list = opt.start_training()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "dPAJ6CGiyyG9"
|
||||||
|
},
|
||||||
"source": [
|
"source": [
|
||||||
"Plot and save data"
|
"\n",
|
||||||
|
"## Evaluation\n",
|
||||||
|
"\n",
|
||||||
|
"Now we can evaluate how our training converged over time. The following graph shows the loss evolution over time in seconds."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 88,
|
"execution_count": 104,
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"colab": {
|
||||||
|
"base_uri": "https://localhost:8080/",
|
||||||
|
"height": 316
|
||||||
|
},
|
||||||
|
"id": "YzHtLIqgyyG9",
|
||||||
|
"outputId": "0388cf17-63e8-4587-a168-362f903e19d5"
|
||||||
|
},
|
||||||
"outputs": [
|
"outputs": [
|
||||||
{
|
{
|
||||||
"name": "stdout",
|
"output_type": "display_data",
|
||||||
"output_type": "stream",
|
|
||||||
"text": [
|
|
||||||
"dict_keys(['HIG'])\n"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"data": {
|
"data": {
|
||||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEWCAYAAABliCz2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAreElEQVR4nO3dd3gVdfr+8feTQAg1KE16lyJNDU1KEFFBRbCsC669YF0RcV1dVxd1XV1XUFEsuCq2FbGgoCAKSKgioCjdAIKE3juEJM/vj3PYXzZfSiBlzknu13XlImfOOTP3cODJ5JnPzMfcHRERKfxigg4gIiIFQwVfRKSIUMEXESkiVPBFRIoIFXwRkSJCBV9EpIhQwRcRKSJU8OWkmFknM1sWdI6cMLM6ZuZmVuwk37/HzOrldS6RgqaCL8dkZqvMrFv25e4+zd0bBZHpSMzsdDP7yMy2mNlOM/vZzO4zs9jcrtvdy7j7yrzImd2R/n7N7AYzm36015hZVTN73czWhX8YrTSzEWbWOAfbu9nMlprZbjPbaGbjzKxslucTzewLM9tuZjvMbLGZPWlmp2TJlhHe7h4z+9XM3jKz0/Pmb0Tykwq+RJUjHaWbWX1gNrAGaO7uCcDvgESgbPbXRzMzqwDMBEoBnQjt31lAMnD+cd6bBPwD6OvuZYEmwIdZnj8HmALMABq7e3mgO5AOtMyyqlnuXgZIALoB+4F5ZtYs93so+UkFX06KmXUxs9Qsj1eZ2f3hI+udZvahmcVnef4SM5sfPmqcaWYtsjz3oJmtCB91Ljazy7I8d4OZzTCz58xsKzDoCHEeA2a6+33uvh7A3Ze5+9XuviPL6/5gZr+Ffwt4OMs22pjZrHC29Wb2kpnFZXnezaxB+PsRZjbMzL4M550d/oFTUAYAu4Br3X2Fh+xw97fc/cXjvLc1oWL9I4C7b3P3t919d/j5Z4C33P0pd98Yfs1v7v43d5+SfWXunhHOcCehHziD8mYXJb+o4EteuorQEWFdoAVwA4CZnQm8CdwGVABeA8aYWYnw+1YQOlpNIFS83zOzqlnW2xZYCVQBnjzCdrsBH+cgX0egEXAe8KiZNQkvzyBUSCsC7cPP33mM9fQJ5zwFWH6UTPmlGzDa3TNP4r2zgQvN7DEz65Dl7x8zK01o3z85yVyfEvoMJYKp4EteGuru69x9GzAWaBVe3g94zd1nh48K3wYOAu0A3P2j8Psy3f1DIAVok2W969z9RXdPd/f9R9huBWB9DvI95u773f0n4CfCbQp3n+fu34XXv4rQD6SkY6xntLt/7+7pwPtZ9vNkfRb+7WKHme0AXj7GaysCGw4/MLNLw+/bbWZfH2sj7j4NuJxQC+hLYKuZDQmf5ziFUD3Iuu5nwuvea2Z/Pc4+rANOPc5rJGAq+JKXNmT5fh9QJvx9bWBgtqJWE6gGYGbXZWn37ACaESpsh605zna3AlWP85qj5guf8P3CzDaY2S5Cfe6KR1rBsdaTnZm9muXk5l+Osb7e7l7+8BfH/u3if/bV3ceE3zMAiDvam7K8fry79yRUnHsR+i3sFmA7kJlt3Q+E1z0aON4Ip+rAtuNtX4Klgi8FYQ3wZNai5u6l3P0DM6sNvA7cDVQIF5iFgGV5//Hu4T0RuCIX+V4BlgIN3b0c8Jds2z8p7n57eIRPGXf/R27XFzYJ6G1mufq/G/5tahIwGWjm7nsJtXwuP8lVXgZMy00myX8q+JITxc0sPsvXiY5nfx243czaWkhpM7s4PBywNKGCvhnAzG4kdIR/Iv4GnGNm/zKz08LraWBm75lZ+Ry8vyyhE6F7wkMb7zjB7RekIYTaL++aWf3w32dZctBWMrNeZtbHzE4Jv68NodbVd+GXPADcFD6JXjn8nhqEzskcaX2xZlbXzF4EuhA6ryERTAVfcmIcoaF3h78Gncib3X0ucCvwEqHWwXLCJ3TdfTEwGJgFbASaExoWeCLrX0HohGMdYJGZ7SR08nEusPsYbz3sfuDq8GtfJ8tQxUjj7lsInfs4AEwnlHk+oR9ax/tBtZ3Q55BC6Afce8C/3P398LqnA12BzsAv4fbaV4SGamYdAdTezPaE1zEFKAe0dvcFud0/yV+mGa9ERIoGHeGLiBQRKvgihYiZ/SHLyKCsX4uCzibBU0tHRKSIOKm7BxaUihUrep06dYKOISISVebNm7fF3StlXx7RBb9OnTrMnTs36BgiIlHFzFYfabl6+CIiRYQKvohIEaGCLyJSRKjgi4gUESr4IiJFRIGN0glPsPAykAZMOXz/DhERKRi5OsI3szfNbJOZLcy2vLuZLTOz5Wb2YHjx5cDH7n4rcGlutisiIicuty2dEYSmtPuv8Ow5w4AeQFOgr5k1BWrw/yeyyMjldo9pyrJNvPvdajIydRWxiMhhuZ1EYSr/d5abNsByd1/p7mnASEIz66QSKvrH3K6Z9TOzuWY2d/PmzSeV64uf1/PIZwu57OUZ/LRmx0mtQ0SksMmPk7bV+d8p6VLDyz4FrjCzVwjNd3pE7j7c3RPdPbFSpf9zZXCO/OvKFgzteyYbdh6g98sz+MvoBezYl3ZS6xIRKSwK7KRteAq1G3PyWjPrCfRs0KDBSW3LzLi0ZTXObVSJ5yemMGLmKr5auIEHuzfmyrNrEBOT69nrRESiTn4c4a8lNEH1YTXCy3LM3ce6e7+EhIRcBSkbX5xHLmnKF3/sSP1KpXngk5+58tWZLFq3M1frFRGJRvlR8OcADcNzXcYBfYAxJ7ICM+tpZsN37sybwtykajlG3daeZ3/XktVb99HzxekMGrOIXQcO5cn6RUSiQW6HZX5AaC7SRmaWamY3u3s6cDcwAVgCjHL3E5p8Ia+O8LNl5cqzazD5/i5c064278xaRddnkxn9YyqaE0BEioKIngAlMTHR8+v2yAvX7uThzxby05odtK17Kk/0bsbpVcrmy7ZERAqSmc1z98TsyyPy1gp53dI5kmbVExh9xzk8dXlzlm3czUUvTOPJLxez52B6vm1TRCRIRfYIP6tte9P414SlfPD9GqqUK8EjlzTl4uZVMdNoHhGJPjrCP4ZTS8fx1OUtGH3nOVQsU4K7//Mj177xPSs27ymQ7YuIFAQd4WeTkem8P3s1/5qwjAOHMri1Uz3+2LUhJeNiCzSHiMjJiqoj/CDFxhjXta/D5IFd6NmyGi9PWUG3IclMWLRBo3lEJKpFZMEv6JbOkVQqW4IhV7Vi1G3tKVOiGLe9O4+bRsxh9da9gWUSEckNtXRy4FBGJm/PXMVz3/zCoUznzi71uT2pPvHF1eYRkcijlk4uFI+N4ZZO9Zh8fxcuPOM0np+YwgXPTeXbpZuCjiYikmMq+CegSrl4Xux7Ju/f0pbiscaNI+bQ7525pG7fF3Q0EZHjisiCHwk9/GPp0KAi4/t35s/dGzMtZQvdhiQz7NvlpKVnBh1NROSo1MPPpbU79vPE2MV8tWgD9SqV5vFLm9GxYcWgY4lIEaYefj6pXr4kr157NiNubE1GpnPNG7O56z8/sGHngaCjiYj8DxX8PNKlUWUm3NuZAd1OZ+LijZw3eAqvT13JoQy1eUQkMqjg56H44rH079aQbwYk0bZeBZ4ct4RLhk5n9sqtQUcTEYnMgh/pJ22Pp1aFUrxxfSKvX5fInoPp/H74dwz4cD6bdqvNIyLB0UnbfLY/LYNh3y5n+NSVlCgWw8ALTueadrUpFhuRP2tFpBDQSduAlIyL5f4LG/HVvZ1oVas8g8Yu5tKXZvDDb9uDjiYiRYwKfgGpV6kM79zUhmFXn8W2vWlc/vJM/vzxz2zbmxZ0NBEpIlTwC5CZcXGLqkwamMRtnevxyQ+pnPvsFN6fvZrMzMhtrYlI4aCCH4DSJYrx0EVNGNe/E41PK8vDoxdy2cszWJAanSepRSQ6qOAH6PQqZRnZrx3P/74Va3cc4NJh0/nrZwvYue9Q0NFEpBCKyIIf7cMyT4SZ0fvM6ky+P4nr29fhP7N/o+vgKXw0d43aPCKSpzQsM8IsXreLRz5fyLzV20msfQpP9G5Gk6rlgo4lIlFEwzKjRNNq5fjotvY8c2ULVm7ZyyUvTuexsYvYfUBtHhHJHRX8CBQTY1yVWJPJA5Po07omI2auouvgZD6fv1bz6orISVPBj2DlS8Xx5GXN+ezODlRNiKf/yPlc/fpsUjbuDjqaiEQhFfwo0LJmeUbf2YEnL2vG4vW76PHCNJ4av4S9B9ODjiYiUUQFP0rExhh/aFubyQOTuPys6ryWvJJuQ5IZt2C92jwikiMq+FGmQpkSPHNlSz65oz3lS8Vx5/s/cN2b3/Prlr1BRxORCKeCH6XOrn0qY+/uwN96NmX+bzu48LmpDP56GfvTMoKOJiIRqsAKvpnVM7M3zOzjgtpmYVcsNoYbO9Rl0v1JXNyiKi9OXs75zyUzcfHGoKOJSATKUcE3szfNbJOZLcy2vLuZLTOz5Wb24LHW4e4r3f3m3ISVI6tcNp7nft+Kkf3aUSoullvemcvNI+awZtu+oKOJSATJ6RH+CKB71gVmFgsMA3oATYG+ZtbUzJqb2RfZvirnaWo5onb1KvDlPZ34y0WNmbVyK92GJDN0UgoHDqnNIyI5LPjuPhXYlm1xG2B5+Mg9DRgJ9HL3Be5+SbavTTkNZGb9zGyumc3dvHlzjndEQorHxtCvc30mDUyiW9MqDPnmF7o/P5XkX/R3KVLU5aaHXx1Yk+VxanjZEZlZBTN7FTjTzB462uvcfbi7J7p7YqVKlXIRr2irmlCSYVefxbs3tyHGjOvf/J473pvHuh37g44mIgEpsJO27r7V3W939/ru/tSxXluU7paZ3zo1rMT4ezvxpwsb8e2yTZw3OJlXpqwgLT0z6GgiUsByU/DXAjWzPK4RXpZr7j7W3fslJCTkxeqKvBLFYrnr3AZ8MyCJTg0r8s+vlnLR0GnMXLEl6GgiUoByU/DnAA3NrK6ZxQF9gDF5EUpH+PmLine truncated
|
|
||||||
"text/plain": [
|
"text/plain": [
|
||||||
"<Figure size 432x288 with 1 Axes>"
|
"<Figure size 432x288 with 1 Axes>"
|
||||||
]
|
],
|
||||||
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5xU9bnH8c+znS3ssvSySJVexd6wRNFw1cQGmqq5xsQaS2JuipqbqLElKuQaE40pgl1jbGii2BtSpAkCIiydhV1Y2GXbc/+YA44rOyyws2dn9vt+vea1c87MOec5HJ3v/H6/M+eYuyMiItKQlLALEBGRlk1BISIiMSkoREQkJgWFiIjEpKAQEZGYFBQiIhKTgkL2i5kdbWaLwq5jT8zMzazfPi4738zGNnFJLc6ejqWZ9Qr+HdOasy4Jn4JCGsXMlpvZifXnu/sb7j4gjJqai7sPcffp8Vi3mU03s+/VmzfWzIobeo+Z5ZnZncEx2WZmK8zscTM7dH9qqX8sGzrmjWVmPczsCTPbaGZlZjbPzL4T9XqGmf3SzBYF+7HKzF4ws5Pq1VBhZlvNrNTM3jazi81Mn13NSN8MJCGZWZq714RdR3Mzs0zgFaAUGA8sBLKAU4LHe+FV9yV/B+YABwA7gGFAl6jXHwe6A98CZgXzjge+CrwU9b7/cvd/m1k+cCxwF3Ao8N24Vi+7KJVlv+zm2+9yM7vGzD4KvkU+YmZZUa+PN7PZUd8Oh0e9dp2ZLQ2+PS4ws69FvfYdM3vLzH5nZiXADbupJdXM/idqHR+aWVHUW040s0+CbU82MwuW62tmr5hZSfDt9yEzK6i3TycGz28ws0fN7G/BNuab2Zgm+udsjG8CPYAz3H2eu9e6+zZ3f9zdb9jdAmb2VzO7OnjePeg+uiSY7mtmm8wsJfpYmtnfgZ7Av8ys3Mx+HLXK84NWzEYz+1mMWg8GHgzqq3H3We7+QrD+E4GvAKe7+3vuXhU8XnT3K3a3Mncvc/dngHOBb5vZ0Mb/s8n+UFBIPJwDjAN6A8OB7wCY2SjgAeD7QHvgj8AzwbdkgKXA0UA+cCPwDzPrGrXeQ4FlQGfgN7vZ7lXAROBUoC1wAbA96vXxRD68hgc1nhzMN+BmoBswCChiN0EU5TTgYaAAeAaYFOO9Te1EYJq7b9uLZV4DxgbPjyXyb3hM1PQb7l4XvYC7fxNYQeTbfK673xr18lHAAOAE4JdmNqiB7b4LTDazCWbWczf78Z67F+9muZjc/X2gmMh/K9IMFBQSD3e7+2p33wT8CxgZzL8I+GPwDbLW3f9KpEviMAB3fyxYrs7dHwE+AQ6JWu9qd78n+HZasZvtfg/4ubsv8og57l4S9fot7l7q7iuAV3fW5e5L3P1ld9/h7huAO4l8gDbkTXd/3t1riXSvjNjLf5/67g5aOaVmVgo8G+O9HYC1OyfMbGSw3JYYA9GvAUcF/frHALcCRwavHRu8vjdudPcKd59DpGupof0/G3gD+AXwadCSPLiB/SgM9qPMzCobUcNqoHAv65Z9pKCQeFgb9Xw7kBs8PwC4ut6HYhGRb/KY2beiuqVKgaFEPlB2WrmH7RYRaZXsVV1m1tnMHg4GU7cA/6i33T2tJ8t2cyZQ0A1WHjzujbG+y929YOeDSMunISXArlaWu88Olvk6kLm7Bdx9KbCNSDAeTSSIVpvZAPYtKBo6vvW3u9ndr3P3IURagbOBp4Muv/r7sSnYj4Ma2o96ugOb9rJu2UcKCmlOK4HfRH8ounu2u081swOAPwGXAu2DD415RLqFdtrTpY5XAn33oa6bgnUPc/e2wDfqbXefuPtNQbdNrrtfvL/rC/wHOMnMcvZyudeAs4AMd18VTH8baEfkA3x3muzS0u6+EbidyJeCQiL7cbCZ9djbdQWtku7Am01Vn8SmoJC9kW5mWVGPvT1r7k/AxWZ2qEXkmNlXzSwPyCHywbQBwMy+S6RFsTf+DPyvmfUP1j/czNo3Yrk8oBwoM7PuwLV7ud3m9DdgDfCUmQ0NBvCzgD0NqL9GJIRfD6anB9NvBl1ou7MO6LOvhZrZb4Ma04Jj/ANgibuXuPtLRLr/ng7+e8gws3SCbsgG1tfWzMYTGR/6h7vP3dfaZO8oKGRvPA9URD1u2JuF3X0G8N9EBn83A0sIBrrdfQFwB/AOkQ+oYcBbe1nfncCjRE6t3ALcD7RpxHI3AqOBMuA54Mm93G6zcfdK4DhgAZFatwCLiAzSnxNj0deIBOLOoHgTyI6a3p2bgZ8HXYHX7EO52cBTRE7lXUak6/G0qNe/RqQb7B/Bez4Fzufzkwx2+peZbSXSYvwZkeOsU2ObkenGRSIiEotaFCIiEpOCQkREYlJQiIhITAoKERGJKSkvCtihQwfv1atX2GWIiCSUDz/8cKO7d6w/PymDolevXsyYMSPsMkREEoqZfba7+ep6EhGRmBQUIiISk4JCRERiUlCIiEhMCgoREYlJQSEiIjEpKEREJCYFRZR/L1jHozP2dBM1EZHWJSl/cLcv3J0p76/gzU82MqRbW4Z0yw+7JBGRFkEtioCZcfvZI2iXk86lU2ZRvqMm7JJERFoEBUWUwpwM7p4wis9KtvGzp+aimzqJiCgovuTQPu350YkH8s/ZqzVeISKCgmK3fnhcP47q14Hrn5nPorVbwy5HRCRUCordSE0x7jx3BLmZ6VwyZSbbqzReISKtl4KiAZ3ysrhrwkiWbijn+n/OD7scEZHQKChiOLJfBy47rh+PfVjMkzOLwy5HRCQUCoo9uPyE/hzSu5CfPz2PJevLwy5HRKTZKSj2IC01hbsnjCIrPZVLp8yksro27JJERJqVgqIRuuRnccc5I/h47Vb+99kFYZcjItKsFBSNdNyATnz/2D489N4Knv1oddjliIg0GwXFXrjmpAGM7lnAdU/M5bOSbWGXIyLSLBQUeyE9NYW7J44iNcW4dMosdtRovEJEkp+CYi/1aJfNbWcNZ+6qMm5+/uOwyxERiTsFxT44aUgXvntkLx58eznT5q8NuxwRkbhSUOyj604ZyLDu+Vz72ByKN28PuxwRkbhRUOyjzLRUJp03Cne4bOosqmvrwi5JRCQuFBT74YD2Odx85jBmrSjl9mmLwi5HRCQuFBT7afzwbpx/aE/++PoyXv14fdjliIg0OQVFE/jF+MEM7JLHVY/OZk1ZRdjliIg0KQVFE8hKT2Xy+aPZUVPHFVNnU6PxChFJIgqKJtK3Yy6/+dpQ3l++ibv+80nY5YiINBkFRRP62qgenDOmB5NeXcKbn2wMuxwRkSahoGhiN5w2hH4dc7nykVms31oZdjkiIvtNQdHEsjPSmHz+aMp31HDlw7OprfOwSxIR2S8tPijMrI+Z3W9mj4ddS2Md2DmPX502lLeXljD51SVhlyMisl/iGhRm9oCZrTezefXmjzOzRWa2xMyui7UOd1/m7hfGs854OHtMD84Y2Y3f/3sx7y4rCbscEZF9Fu8WxYPAuOgZZpYKTAZOAQYDE81ssJkNM7Nn6z06xbm+uDEzfv21YfRqn8MVD8+ipHxH2CWJiOyTuAaFu78ObKo3+xBgSdBSqAIeBk5397nuPr7eI6F/6pybmcY9541i8/Zqrnp0DnUarxCRBBTGGEV3YGXUdHEwb7fMrL2Z3QuMMrOfxnjfRWY2w8xmbNiwoemq3U9DuuXzi/GDeW3xBu57Y1nY5YiI7LW0sAvYE3cvAS5uxPvuA+4DGDNmTIv66v6NQ3vyztKN3DZtEQf3asdBBxSGXZKISKOF0aJYBRRFTfcI5iUtM+OWM4fTrSCLy6fOpnR7VdgliYg0WhhB8QHQ38x6m1kGMAF4JoQ6mlXbrHQmTRzN+q2VXPPYR7i3qEaPiEiD4n167FTgHWCAmRWb2YXuXgNcCkwDFgKPuvv8eNbRUowoKuC6Uwbx74Xr+Mtby8MuR0SkUeI6RuHuExuY/zzwfDy33VJdcGQv3llaws0vLOSgA9oxoqgg7JJERGJq8b/MTjZmxu1nD6djbiaXTp3JlsrqsEsSEYlJQRGCguwM7jlvFKtLK7nuCY1XiEjLpqAIyUEHFHLNSQN4fu5a/vHeirDLERFpkIIiRN8/pg/HHtiR/312AfNXl4VdjojIbikoQpSSYtx5zgjaZadz2ZRZlO+oCbskEZEvUVCErH1uJndNGMXykm38/Km5Gq8QkRZHQdECHNanPVeeeCBPz17NYzOKwy5HROQLFBQtxCXH9eOIvu355TPzWLxua9jliIjsoqBoIVJTjN9PGEluZhqXPDST7VUarxCRlkFB0YJ0ysvi9+eOYsmGcm54plVc1UREEoCCooU5qn8HLhnbj0dnFPPULI1XiEj4FBQt0JUn9ueQXoX87Kl5LN1QHnY5ItLKKShaoLTUFO6aOJLMtBQueWgmldW1YZckIq2YgqKF6prfhjvOGcHHa7fy6+cWhF2OiLRiCooW7PiBnbnomD78490VPPvR6rDLEZFWSkHRwl178gBG9Szgp0/M5bOSbWGXIyKtkIKihUtPTeHuCaMwg0unzGJHjcYrRKR5KSgSQFFhNredPYK5q8q45YWPwy5HRFoZBUWCOHlIF75zRC/+8tZyps1fG3Y5ItKKKCgSyE9PHcjQ7m259rE5FG/eHnY5ItJKKCgSSGZaKpMmjqbO4bKps6iurQu7JBFpBRQUCaZXhxxuOXMYs1aUcvtLi8IuR0RaAQVFAho/vBvnHdqTP762jFcXrQ+7HBFJcgqKBPXL8YMZ2CWPqx+dw9qyyrDLEZEkpqBIUFnpqUw6bzSV1bVcPnUWNRqvEJE4UVAksH6dcvn1GUN5f/km7v7PJ2GXIyJJSkGR4L4+ugdnHdSDe15dwpufbAy7HBFJQgqKJPCr04fQt2MuVz4ym/VbNV4hIk1LQZEEsjPSmHzeaLZWVvOjR2ZTW+dhlyQiSURBkSQGdMnjxLine truncated
|
||||||
},
|
},
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"needs_background": "light"
|
"needs_background": "light"
|
||||||
},
|
}
|
||||||
"output_type": "display_data"
|
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"source": [
|
"source": [
|
||||||
"import matplotlib.pyplot as plt\n",
|
"import matplotlib.pyplot as plt\n",
|
||||||
"\n",
|
"\n",
|
||||||
"wct_times = {}\n",
|
"plt.plot(np.cumsum(time_list),loss_list)\n",
|
||||||
"wct_times[mode] = wct_time\n",
|
|
||||||
"print(format(wct_times.keys()))\n",
|
|
||||||
"\n",
|
|
||||||
"wct_time = np.cumsum(time_list)\n",
|
|
||||||
"plt.plot(wct_time,loss_list)\n",
|
|
||||||
"plt.yscale('log')\n",
|
"plt.yscale('log')\n",
|
||||||
"plt.xlabel('Wall clock time')\n",
|
"plt.xlabel('Wall clock time [s]'); plt.ylabel('Loss')\n",
|
||||||
"plt.xlabel('Loss')\n",
|
"plt.title('Linear chain - '+MODE+' with '+str(OPTIMIZER.__name__))\n",
|
||||||
"plt.title('Linear Chain - '+mode+'_'+str(optimizer.__name__))\n",
|
"plt.show()"
|
||||||
"plt.show()\n"
|
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"source": [
|
||||||
|
"For all three methods, you'll see a big linear step right at the start. As we're -- for fairness -- measuring the whole runtime, this first step includes all TensorFlow initialization steps, which are significantly more involved for HIG and SIP. Adam is much faster in terms of initialization, and likewise faster per training iteration.\n",
|
||||||
|
"\n",
|
||||||
|
"All three methods by themselves manage to bring down the loss. What's more interesting is to see how they compare. For this, the next cell stores the training evolution, and this notebook needs to be run one time with each of the three methods to produce the final comparison graph."
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"id": "xnr9BvlDU8H6"
|
||||||
|
}
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 89,
|
"execution_count": 105,
|
||||||
"metadata": {},
|
"metadata": {
|
||||||
|
"id": "bZMVQaJiyyG-"
|
||||||
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"path = './'\n",
|
"path = '/home/'\n",
|
||||||
"name1 = 'time'+mode\n",
|
"namet = 'time'\n",
|
||||||
"name2 = 'loss'+mode\n",
|
"namel = 'loss'\n",
|
||||||
"np.savetxt(path+name1+'.txt',time_list)\n",
|
"np.savetxt(path+MODE+namet+'.txt',time_list)\n",
|
||||||
"np.savetxt(path+name2+'.txt',loss_list)"
|
"np.savetxt(path+MODE+namel+'.txt',loss_list)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"source": [
|
||||||
|
"After runs with each of the methods, we can show them side by side:"
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"id": "KS1lL5LsV9JV"
|
||||||
|
}
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"source": [
|
||||||
"metadata": {},
|
"from os.path import exists\n",
|
||||||
"outputs": [],
|
"\n",
|
||||||
"source": []
|
"# if previous runs are available, compare all 3\n",
|
||||||
|
"if exists(path+'HIG'+namet+'.txt') and exists(path+'HIG'+namel+'.txt') and exists(path+'SIP'+namet+'.txt') and exists(path+'SIP'+namel+'.txt') and exists(path+'GD'+namet+'.txt') and exists(path+'GD'+namel+'.txt'):\n",
|
||||||
|
" lt_hig = np.loadtxt(path+'HIG'+namet+'.txt')\n",
|
||||||
|
" ll_hig = np.loadtxt(path+'HIG'+namel+'.txt')\n",
|
||||||
|
"\n",
|
||||||
|
" lt_sip = np.loadtxt(path+'SIP'+namet+'.txt')\n",
|
||||||
|
" ll_sip = np.loadtxt(path+'SIP'+namel+'.txt')\n",
|
||||||
|
"\n",
|
||||||
|
" lt_gd = np.loadtxt(path+'GD'+namet+'.txt')\n",
|
||||||
|
" ll_gd = np.loadtxt(path+'GD'+namel+'.txt')\n",
|
||||||
|
"\n",
|
||||||
|
" plt.plot(np.cumsum(lt_gd),ll_gd, label=\"GD\")\n",
|
||||||
|
" plt.plot(np.cumsum(lt_sip),ll_sip, label=\"SIP\")\n",
|
||||||
|
" plt.plot(np.cumsum(lt_hig),ll_hig, label=\"HIG\")\n",
|
||||||
|
" plt.yscale('log')\n",
|
||||||
|
" plt.xlabel('Wall clock time [s]'); plt.ylabel('Training loss'); plt.legend()\n",
|
||||||
|
" plt.title('Linear chain comparison ')\n",
|
||||||
|
" plt.show()\n",
|
||||||
|
"else:\n",
|
||||||
|
" print(\"Run this notebook three times with MODE='HIG|SIP|GD' to produce the final graph\")\n"
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"colab": {
|
||||||
|
"base_uri": "https://localhost:8080/",
|
||||||
|
"height": 316
|
||||||
|
},
|
||||||
|
"id": "OsKQ0MyX6Zmx",
|
||||||
|
"outputId": "c76d6002-24e3-4f7d-e3a2-042ce15f9478"
|
||||||
|
},
|
||||||
|
"execution_count": 106,
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"output_type": "display_data",
|
||||||
|
"data": {
|
||||||
|
"text/plain": [
|
||||||
|
"<Figure size 432x288 with 1 Axes>"
|
||||||
|
],
|
||||||
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3hUZdrH8e89k94rJCGEXgWkg6iAooJiW3VFV9e+9tXVtZe1rRV27Yu9r93VV7GLIoKoNAXpHUII6b1nnvePMwlDSCWTnGRyf67rXDNzzpkz98lAfnme5xQxxqCUUko1xGF3AUoppTo2DQqllFKN0qBQSinVKA0KpZRSjdKgUEop1SgNCqWUUo3SoFBtRkSOFJENdtfRFBExItL/IN+7RkSmermkDktEzhGRr+yuQ7Uv0fMoVGuJyHbgEmPMN3bXcjBExAADjDGb7a5FqY5IWxTK54iIn901+CL9uXZdGhSqzYjIVBFJ9Xi9XURuEJFVIpIvIu+ISJDH8hNF5FcRyRORH0VkhMeyW0Rki4gUishaEfmDx7ILRGSxiDwqItnA3fXU4hSR2zy2sVxEenqscoyIbHJ/9tMiIu739RORb0UkW0SyROS/IhJVZ5+OcT+/W0TeFZHX3J+xRkTGNvLzOUREvhaRHBHZKyK3uecHishjIpLmnh4TkUDPn6mI3CQiGSKyR0ROFZETRGSje1u3eXzG3SLyvvtnXSgiK0Tk0IP9ubrnLXIvF/eyDBEpEJHVIjLMvSzS/XPIFJEdInKHiDg8trtIROaISK6IbBOR4xv6OSn7aVCo9nYmMAPoA4wALgAQkVHAS8BlQCzwLPBxzS9IYAtwJBAJ3AO8ISKJHtudAGwFugP31/O51wNnAycAEcBFQInH8hOBce6azgSmu+cL8CCQBAwBelJPEHk4GXgbiAI+Bp6qbyURCQe+Ab5wb7s/MN+9+HZgIjASOBQYD9zh8fYEIAjoAfwDeB44FxiD9TO6U0T6eKx/CvAeEAO8CXwkIv7uZa35uR4HTAYGut9/JpDtXvake15fYApwHnBhne1uAOKAR4AXa8JZdUDGGJ10atUEbAeOqWf+VCC1znrnerx+BHjG/XwucF+d928ApjTwmb8Cp7ifXwDsbKLGDTXr17PMAEd4vH4XuKWBdU8FVta371gB8o3HsqFAaQPbOdtzO3WWbQFO8Hg9Hdju8TMtBZzu1+Hu+id4rL8cONWjpp88ljmAPcCRB/Nzdc9b5H5+NLARK9QcHus4gQpgqMe8y4AFHtvY7LEsxL0PCXb/W9ap/klbFKq9pXs8LwHC3M97AX93d/3kiUge1l/vSQAicp5Ht1QeMAzrr9Eau5r43J5Yv4BbVJeIdBeRt0Vkt4gUAG/U+dymthPUQN9+Y/UkATs8Xu9wz6uRbYypdj8vdT/u9Vheyr6fK3j8bIwxLiAVL/xcjTHfYrWYngYyROQ5EYlwv9+/nn3o4fE63WM7NS07z5pVB6JBoTqKXcD9xpgojynEGPOWiPTC6l65Gog1xkQBv2N1C9Vo6vC9XUC/g6jrAfe2hxtjIrC6eLzRRbILq1umPmlYwVkjxT3vYNWOxbjHCZKBNG/8XI0xTxhjxmC1ngYCNwJZQGU9+7C7FfugbKRBobzFX0SCPKaWHiHzPHC5iExwD5KGishMd19+KNYvrEwAEbkQ6y/flngBuE9EBri3P0JEYpvxvnCgCMgXkR5Yvwi9YR6QKCJ/cw9eh4vIBPeyt4A7RCReROKwxiHeaMVnjRGR09zfyd+AcuAnWvlzFZFx7u/LHygGygCXu7XzLnC/e796YY0RtWYflI00KJS3fIbV5VEz3d2SNxtjlgF/werKyAU24x7oNsasBf4FLMHqYhkOLG5hff/G+uX1FVAAvAgEN+N99wCjgXzgU+B/LfzcehljCoFjgZOwumE2AUe5F/8TWAasAlYDK9zzDtb/AbOwfq5/Bk4zxlR64ecagRXwuVhdS9nAbPeyv2KFx1ZgEdYg+kut2AdlIz3hTikfJiJ3A/2NMefaXYvqvLRFoZRSqlEaFEoppRqlXU9KKaUapS0KpZRSjfLJi3zFxcWZ3r17212GUkp1KsuXL88yxsTXne+TQdG7d2+WLVtmdxlKKdWpiMiO+uZr15NSSqlGaVAopZRqlAaFUkqpRvnkGIVSSnlDZWUlqamplJWV2V2KVwUFBZGcnIy/v3/TK6NBoZRSDUpNTSU8PJzevXvjK/dVMsaQnZ1Namoqffr0afoNaNeTUko1qKysjNjYWJ8JCQARITY2tkWtJA0KpZRqhC+FRI2W7pMGhYcFuxbw4aYP7S5DKaU6FA0KN2MM7218j/t+uo/1OevtLkcppWrt3buXP/3pT/Tt25cxY8Zw2GGH8eGHH7JgwQIiIyMZNWoUgwYNYvLkycybN8/rn69B4SYi/PPwfxIdGM0N399AcWWx3SUppRTGGE499VQmT57M1q1bWb58OW+//TapqakAHHnkkaxcuZINGzbwxBNPcPXVVzN//nyv1qBB4SE6KJqHJz/MrsJd3LvkXvTKukopu3377bcEBARw+eWX187r1asXf/3rXw9Yd+TIkfzjH//gqaee8moNenish/XpBSQFHcKVh17JU78+xYTECZw24DS7y1JKdQD3fLKGtWkFXt3m0KQI7jrpkEbXWbNmDaNHj272NkePHs3s2bObXrEFNCg87Hj1MrqVbOLI2GSWxsXy4JJ7GZ6bzoDuoyAiCcITwS/A7jKVUl3YVVddxaJFiwgICKg3ENqiJ0SDwsPEQw8hfVU6zqwN3JWfx5+Tornhtyd4Ky2dkJoffmg3KzQiergfEz2e97DCJCDE3h1RSnldU3/5t5VDDjmEDz74oPb1008/TVZWFmPHjq13/ZUrVzJkyBCv1qBB4SFyxh1EzriD33blcdu8tWTv/oHslDe48ZCTeKrPMUjhHijYDQV7IHc77FgMZXkHbig4el94hHsGiUfABEW0+/4ppTqfo48+mttuu425c+dyxRVXAFBSUlLvuqtWreK+++7jhRde8GoNGhT1OLRnFO9cfhhfre3Lnd9nsVC+YOavo5l9/BUckhS5/8oVxVZwFOyGgrR9jzWhkrYSijMP/JCAcHdrJKn+IInoYQWOD57so5RqPhHho48+4rrrruORRx4hPj6e0NBQHn74YQB++OEHRo0aRUlJCd26deOJJ55g2rRp3q3BF4/sGTt2rPHWjYtKKio4/cPzSC3ZRPH2qzlt2GhuOG4QCZFBzd9IVbk7ONI8wsQzXNKgKB2Ma//3+QU13CKpmUK7gUMPXlOqLaxbt87r3TgdRX37JiLLjTEH9Glpi6IJIQEBvHLi45zxyRlEDfiAj3+LZd6qNC49si+XTelHaGAzfoR+gRDd25oaUl0FRXvdrZG0/VsnBWmw6ycrXFyV+7/P4ecOk3qCJLzmMQGczbtKpFJK1aVB0QzdQ7vzwBEPcOX8K/nDMcspTjuFJ77dzJu/7OLvxw3kzLE9cTpa2UXk9IPIHtbUEJcLSrLqtEw8nu9ZBRu+gKrSOm8UCOvecKukJlT8W9BKUkp1GRoUzXRk8pFcOOxCXv79ZWZPOYyLjpjE/Z+u49b/reaVxdu5beYQpgw84J7k3uVwQFg3a0oaWf86xlgD7AeEifsxezNs+wHK8w98b0js/kdv1dflFRjWtvuolOpwNCha4K+j/sqKvSu4+8e7ee/E93j/8sP44vd0HvpiPee/9AtHDojj9plDGJxg4xFNItYgeHA0dG/kcL7ywgPHSTxDJXUplGQf+L7ASI/DgusMvteESlCUDsIr5UN0MLuF0orS+OMnfyQ5PJnXj3+dAGcAFVUuXluynSe/3UxhWSV/HNOT648bSPeITt6VU1lazyB82v5T0V6gzr8h/xCPcZMGBuJD4nQQXnV4Opht0RZFCyWFJXHf4fdx7XfX8u/l/+aW8bcQ4OfgkiP7csaYZJ78djOvLdnOx7+lcdmUvlw6uS8hAZ30x+wfDDF9rakh1ZVQmL4vSGqDxR0qOxZb81xV+7/P4V/nZEWPIKkZhA/rbo3dKKVspf8LD8LRKUdz7pBzeWPdG4xLGMe0FOuY5aiQAO48cSjnHdaLh79Yz2PfbOLNn3dyw/RB/HFMsk/eAAWnP0T1tKaGuKqtc0lqDwuu0zrZvQLWzYPq8v3fJ06Yfj9MvKJt90GpDu7+++/nzTffxOl04nA4ePbZZ7n55puZM2cOY8eOpXfv3oSHhyMiJCQk8Nprr5GQkOC1z9egOEjXjbmOFRkruHPxnQyJGUJSWFLtsl6xofznnDEs35HDffPWcdP7q+gVE8KEvrE2Vmwjh9M6RDc8ARo6qMsYKMnZ/9Dg7x6EnT9pUKgubcmSJcybN48VK1YQGBhIVlYWFRUVB6z33XffERcXx2233cYDDzzAE0884bUatJP4IAU4A5gzeQ7GGG5ceCOVdc9vAMb0iuHJs0cBsCOn/lPulZsIhMZCwnAYOB3GXmR1eZXm2F2ZUrbas2cPcXFxBAYGAhAXF0dSUlKD60+ePJnNmzd7tQZtUbRCz4ie3DXpLm78/kaeXPEk14+9/oB1YsOsq81mFx34F4BqQkgM5O2yuwqlLJ/fAumrvbvNhOFw/EONrnLcccdx7733MnDgQI455hhmzZrFlClTGlx/3rx5DB8+3KtlaouilWb0nsGZA8/k5TUvszB14QHLQwL8CPJ3kFNcXs+7VaNCYuo/RFepLiQsLIzly5fz3HPPER8fz6xZs3jllVcOWO+oo45i5MiRFBQUcOutt3q1Bm1ReMFN42/i18xfuX3R7bx30nskhO4/iBQbGqgtioMRHGN1PRmj52Uo+zXxl39bcjqdTJ06lalTpzJ8+HBeffXVA9apGaNoC9qi8IJAZyBzpsyhvLqcmxfeTFWdQ0G7RQSyt7DMpuo6sZBYqCqDSh3fUV3XhLine truncated
|
||||||
|
},
|
||||||
|
"metadata": {
|
||||||
|
"needs_background": "light"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"source": [
|
||||||
|
"This graph makes the significant differences in terms of convergence very clear: Adam (the blue `GD` curve), performs a large number of updates, but its rough approximation of the Hessian is not enough to converge to high accuracies. It stagnates at a high loss level.\n",
|
||||||
|
"\n",
|
||||||
|
"The SIP updates don't outperform Adam in this scenario. This is caused by the relatively simple phyiscs (the _linear_ oscillators), and the higher runtime cost of SIP. If you run this example longer, SIP will acutally overtake Adam, but start suffering from numerical issues with the full inversion.\n",
|
||||||
|
"\n",
|
||||||
|
"The HIGs perform much better than the two others: despite being fairly slow per iteration, the half-inversion produces a very good update, that makes the training converge to very low loss values very quickly. The HIGs reach an accuracy that is around four order of magnitudes better than the other two methods."
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"id": "XifTlo4xWPDb"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"source": [
|
||||||
|
"## Next steps\n",
|
||||||
|
"\n",
|
||||||
|
"There's a variety of interesting avenues for further tests and modifications with this notebook:\n",
|
||||||
|
"\n",
|
||||||
|
"* Most importantly, we've acutally only looked at training performance so far! This keeps the notebook reasonably short, but it's admittedly bad practice. While we can claim that HIGs likewise work on _real_ test data, this is a great next step: allocate proper test samples, and re-run the evaluation for all three methods on the test data.\n",
|
||||||
|
"* Also, you can vary the physics behavior: use more oscillators for longer or shorter time spans, or even include a non-linear force (as employed in the HIG paper). Be warned: for the latter, the SIP version will require a new inverse solver, though."
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"id": "UiaLlPVz-clP"
|
||||||
|
}
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"metadata": {
|
"metadata": {
|
||||||
@@ -635,8 +793,7 @@
|
|||||||
"hash": "60a46218545510a486eb79ffac0761a327b44b02462576ac242e6c15f2abc4b7"
|
"hash": "60a46218545510a486eb79ffac0761a327b44b02462576ac242e6c15f2abc4b7"
|
||||||
},
|
},
|
||||||
"kernelspec": {
|
"kernelspec": {
|
||||||
"display_name": "Python 3",
|
"display_name": "Python 3.8.10 64-bit ('tf250': conda)",
|
||||||
"language": "python",
|
|
||||||
"name": "python3"
|
"name": "python3"
|
||||||
},
|
},
|
||||||
"language_info": {
|
"language_info": {
|
||||||
@@ -649,9 +806,15 @@
|
|||||||
"name": "python",
|
"name": "python",
|
||||||
"nbconvert_exporter": "python",
|
"nbconvert_exporter": "python",
|
||||||
"pygments_lexer": "ipython3",
|
"pygments_lexer": "ipython3",
|
||||||
"version": "3.8.5"
|
"version": "3.8.10"
|
||||||
|
},
|
||||||
|
"orig_nbformat": 4,
|
||||||
|
"colab": {
|
||||||
|
"name": "LinearChain_GD_PG_HIG-fromSlack-colab.ipynb",
|
||||||
|
"provenance": [],
|
||||||
|
"collapsed_sections": []
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"nbformat": 4,
|
"nbformat": 4,
|
||||||
"nbformat_minor": 2
|
"nbformat_minor": 0
|
||||||
}
|
}
|
||||||
Reference in new issue
Block a user