diff --git a/_toc.yml b/_toc.yml index b0ab3be..f12ce99 100644 --- a/_toc.yml +++ b/_toc.yml @@ -34,6 +34,15 @@ parts: chapters: - file: reinflearn-intro.md - file: reinflearn-code.ipynb +- caption: Improved Gradients + chapters: + - file: physgrad.md + - file: physgrad-comparison.ipynb + - file: physgrad-nn.md + - file: physgrad-code.ipynb + - file: physgrad-hig.md + - file: physgrad-hig-code.ipynb + - file: physgrad-discuss.md - caption: PBDL and Uncertainty chapters: - file: bayesian-intro.md diff --git a/json-cleanup-for-pdf.py b/json-cleanup-for-pdf.py index a7e820c..afcaaa2 100644 --- a/json-cleanup-for-pdf.py +++ b/json-cleanup-for-pdf.py @@ -17,10 +17,10 @@ fileList = [ "bayesian-code.ipynb", "supervised-airfoils.ipynb", # pytorch "reinflearn-code.ipynb", # phiflow "physgrad-comparison.ipynb", # jax + "physgrad-code.ipynb", # pip ] -#fileList = [ "diffphys-code-burgers.ipynb"] # debug, only 1 file -#fileList = [ "diffphys-code-ns.ipynb"] # debug, only 1 file +#fileList = [ "physgrad-code.ipynb"] # debug, only 1 file # main @@ -55,6 +55,10 @@ for fnOut in fileList: res.append( re.compile(r"Building wheel") ) # phiflow install, also gives weird unicode characters res.append( re.compile(r"warnings.warn") ) # phiflow warnings res.append( re.compile(r"WARNING:absl") ) # jax warnings + + res.append( re.compile(r"ERROR: pip") ) # pip dependencies + res.append( re.compile(r"requires imgaug") ) # pip dependencies + # remove all "warnings.warn" from phiflow? # shorten data line: "0.008612174447657694, 0.02584669669548606, 0.043136357266407785" diff --git a/make-pdf.sh b/make-pdf.sh index fc171b4..5a3daca 100755 --- a/make-pdf.sh +++ b/make-pdf.sh @@ -1,16 +1,14 @@ # source this file with "." in a shell -# note this script assumes the following paths/versions -# python3.7 -# /Users/thuerey/Library/Python/3.7/bin/jupyter-book +# note this script assumes the following paths/versions: python3.7 , /Users/thuerey/Library/Python/3.7/bin/jupyter-book + +# do clean git checkout for changes from json-cleanup-for-pdf.py via: +# git checkout diffphys-code-burgers.ipynb diffphys-code-ns.ipynb diffphys-code-sol.ipynb physicalloss-code.ipynb bayesian-code.ipynb supervised-airfoils.ipynb reinflearn-code.ipynb echo echo WARNING - still requires one manual quit of first pdf/latex pass, use shift-x to quit echo -# do clean git checkout for changes from json-cleanup-for-pdf.py? -# git checkout diffphys-code-burgers.ipynb diffphys-code-ns.ipynb diffphys-code-sol.ipynb physicalloss-code.ipynb bayesian-code.ipynb supervised-airfoils.ipynb reinflearn-code.ipynb - # warning - modifies notebooks! python3.7 json-cleanup-for-pdf.py diff --git a/physgrad-code.ipynb b/physgrad-code.ipynb index 3be0019..0a9e8c7 100644 --- a/physgrad-code.ipynb +++ b/physgrad-code.ipynb @@ -1,641 +1,651 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Practical Example with SIP Training\n", - "\n", - "placeholder only, insert more complex SIP example \n", - "\n", - "..." - ] - }, - { - "cell_type": "code", - "execution_count": 77, - "metadata": {}, - "outputs": [], - "source": [ - "dont run !!! import numpy as np\n", - "import tensorflow as tf\n", - "import time\n", - "import os\n", - "\n", - "os.environ[\"CUDA_VISIBLE_DEVICES\"]='' # set GPUs\n", - "#tf.config.run_functions_eagerly(True) # deactivate Tensorflow Tracing" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Physics Model\n", - "\n", - "Differential equation: Coupled Linear Oscillators with Control term\n", - "\n", - "Solver: Runge-Kutta-4 \n", - "\n", - "Nx: Number of oscillators,\n", - "Nt: Number of time evolution steps,\n", - "dt: Length of one time step" - ] - }, - { - "cell_type": "code", - "execution_count": 78, - "metadata": {}, - "outputs": [], - "source": [ - "Nx = 2\n", - "Nt = 24\n", - "dt = 0.5\n", - "\n", - "def build_laplace(n,boundary='0'):\n", - " if n==1:\n", - " return np.zeros((1,1),dtype=np.float32)\n", - " d1 = -2 * np.ones((n,),dtype=np.float32)\n", - " d2 = 1 * np.ones((n-1,),dtype=np.float32)\n", - " lap = np.zeros((n,n),dtype=np.float32)\n", - " lap[range(n),range(n)]=d1\n", - " lap[range(1,n),range(n-1)]=d2\n", - " lap[range(n-1),range(1,n)]=d2\n", - " if boundary=='0':\n", - " lap[0,0]=lap[n-1,n-1]=-1\n", - "\n", - " return lap\n", - "\n", - "@tf.function\n", - "def coupled_oscillators_batch( x, control):\n", - " print('coup harm osc tracing')\n", - " '''\n", - " ODE of type: x' = f(x)\n", - " :param x_in: shape: (batch, 2 * number of osc)\n", - " order second index: x_i , v_i\n", - " :param control: shape:(batch,)\n", - " :return:\n", - " '''\n", - " n_osc = x.shape[1]//2\n", - "\n", - " # natural time evo\n", - " a1 = np.array([[0,1],[-1,0]],dtype=np.float32)\n", - " a2 = np.eye(n_osc,dtype=np.float32)\n", - " A = np.kron(a1,a2)\n", - " x_dot1 = tf.tensordot(x,A,axes = (1,1))\n", - "\n", - " # interaction term\n", - " interaction_strength = 0.2\n", - " b1 = np.array([[0,0],[1,0]],dtype=np.float32)\n", - " b2 = build_laplace(n_osc)\n", - " B = interaction_strength * np.kron(b1,b2)\n", - " x_dot2 = tf.tensordot(x,B, axes=(1, 1))\n", - "\n", - " # control term\n", - " control_vector = np.zeros((n_osc,),dtype=np.float32)\n", - " control_vector[-1] = 1.0\n", - " c1 = np.array([0,1],dtype=np.float32)\n", - " c2 = control_vector\n", - " C = np.kron(c1,c2)\n", - " x_dot3 = tf.tensordot(control,C, axes=0)\n", - "\n", - " #all terms\n", - " x_dot = x_dot1 + x_dot2 +x_dot3\n", - " return x_dot\n", - "\n", - "@tf.function\n", - "def runge_kutta_4_batch(x_0, dt, control, ODE_f_batch):\n", - "\n", - " f_0_0 = ODE_f_batch(x_0, control)\n", - " x_14 = x_0 + 0.5 * dt * f_0_0\n", - "\n", - " f_12_14 = ODE_f_batch(x_14, control)\n", - " x_12 = x_0 + 0.5 * dt * f_12_14\n", - "\n", - " f_12_12 = ODE_f_batch(x_12, control)\n", - " x_34 = x_0 + dt * f_12_12\n", - "\n", - " terms = f_0_0 + 2 * f_12_14 + 2 * f_12_12 + ODE_f_batch(x_34, control)\n", - " x1 = x_0 + dt * terms / 6\n", - "\n", - " return x1\n", - "\n", - "@tf.function\n", - "def solver(x0, control):\n", - " x = x0\n", - " for i in range(Nt):\n", - " x = runge_kutta_4_batch(x, dt, control[:,i], coupled_oscillators_batch)\n", - " return x\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Network Model" - ] - }, - { - "cell_type": "code", - "execution_count": 79, - "metadata": {}, - "outputs": [], - "source": [ - "act = tf.keras.activations.tanh\n", - "model = tf.keras.models.Sequential([\n", - " tf.keras.layers.InputLayer(input_shape=(2*Nx)),\n", - " tf.keras.layers.Dense(20, activation=act),\n", - " tf.keras.layers.Dense(20, activation=act),\n", - " tf.keras.layers.Dense(20, activation=act),\n", - " tf.keras.layers.Dense(Nt, activation='linear')\n", - " ])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Loss Function" - ] - }, - { - "cell_type": "code", - "execution_count": 80, - "metadata": {}, - "outputs": [], - "source": [ - "@tf.function\n", - "def loss_function(a,b):\n", - " diff = a-b\n", - " loss_batch = tf.reduce_sum(diff**2,axis=1)\n", - " loss = tf.reduce_sum(loss_batch)\n", - " return loss" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Data set\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 81, - "metadata": {}, - "outputs": [], - "source": [ - "N = 2**12\n", - "x_train = np.random.rand(N, 2 * Nx).astype(np.float32)\n", - "y_train = x_train" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Optimization\n", - "For the optimization procedure, we need to set up some parameters. \n", - "1. Adam: The truncation parameter has no meaning here.\n", - "2. PG: The specified optimizer is the one used for network optimization. The physics inversion is done via Gauss-Newton and corresponds to an exact inversion since the physical optimization landscape is quadratic. For the Jacobian inversion in Gauss-Newton, we can specify a truncation parameter.\n", - "3. HIG: To receive the HIG algorithm, the optimizer has to be set to SGD. For the Jacobian half-inversion, we can specify a truncation parameter. Optimal batch sizes are typically lower. Learning rates can usually be chosen around 1.\n", - "\n", - "Specify the maximal simulation time in seconds via max_number." - ] - }, - { - "cell_type": "code", - "execution_count": 82, - "metadata": {}, - "outputs": [], - "source": [ - "# Adam Training\n", - "if 0:\n", - " mode = 'GD' \n", - " optimizer = tf.keras.optimizers.Adam\n", - " batch_size = 256\n", - " learning_rate = 0.001\n", - " max_number = 30#*60\n", - " trunc = 10**-10 " - ] - }, - { - "cell_type": "code", - "execution_count": 83, - "metadata": {}, - "outputs": [], - "source": [ - "# PG training\n", - "if 0:\n", - " mode = 'PG' \n", - " optimizer = tf.keras.optimizers.Adam\n", - " batch_size = 256\n", - " learning_rate = 0.001\n", - " max_number = 30#*60\n", - " trunc = 10**-10" - ] - }, - { - "cell_type": "code", - "execution_count": 84, - "metadata": {}, - "outputs": [], - "source": [ - "# HIG training\n", - "if 1:\n", - " mode = 'HIG' # GD, PG, HIG\n", - " optimizer = tf.keras.optimizers.SGD\n", - " batch_size = 32\n", - " learning_rate = 1.0\n", - " max_number = 30#*60\n", - " trunc = 10**-10" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Function to construct the half-inverse of matrix for HIGs" - ] - }, - { - "cell_type": "code", - "execution_count": 85, - "metadata": {}, - "outputs": [], - "source": [ - "from tensorflow.python.framework import ops\n", - "from tensorflow.python.framework import tensor_shape\n", - "from tensorflow.python.ops import array_ops\n", - "from tensorflow.python.ops import math_ops\n", - "from tensorflow.python.util import dispatch\n", - "from tensorflow.python.util.tf_export import tf_export\n", - "from tensorflow.python.ops.linalg.linalg_impl import _maybe_validate_matrix,svd\n", - "\n", - "@tf_export('linalg.gpinv')\n", - "@dispatch.add_dispatch_support\n", - "def gpinv(a, rcond=None,beta=0.5, validate_args=False, name=None):\n", - "\n", - " with ops.name_scope(name or 'pinv'):\n", - " a = ops.convert_to_tensor(a, name='a')\n", - "\n", - " assertions = _maybe_validate_matrix(a, validate_args)\n", - " if assertions:\n", - " with ops.control_dependencies(assertions):\n", - " a = array_ops.identity(a)\n", - "\n", - " dtype = a.dtype.as_numpy_dtype\n", - "\n", - " if rcond is None:\n", - "\n", - " def get_dim_size(dim):\n", - " dim_val = tensor_shape.dimension_value(a.shape[dim])\n", - " if dim_val is not None:\n", - " return dim_val\n", - " return array_ops.shape(a)[dim]\n", - "\n", - " num_rows = get_dim_size(-2)\n", - " num_cols = get_dim_size(-1)\n", - " if isinstance(num_rows, int) and isinstance(num_cols, int):\n", - " max_rows_cols = float(max(num_rows, num_cols))\n", - " else:\n", - " max_rows_cols = math_ops.cast(\n", - " math_ops.maximum(num_rows, num_cols), dtype)\n", - " rcond = 10. * max_rows_cols * np.finfo(dtype).eps\n", - "\n", - " rcond = ops.convert_to_tensor(rcond, dtype=dtype, name='rcond')\n", - "\n", - " # Calculate pseudo inverse via SVD.\n", - " # Note: if a is Hermitian then u == v. (We might observe additional\n", - " # performance by explicitly setting `v = u` in such cases.)\n", - " [\n", - " singular_values, # Sigma\n", - " left_singular_vectors, # U\n", - " right_singular_vectors, # V\n", - " ] = svd(\n", - " a, full_matrices=False, compute_uv=True)\n", - "\n", - " # Saturate small singular values to inf. This has the effect of make\n", - " # `1. / s = 0.` while not resulting in `NaN` gradients.\n", - " cutoff = rcond * math_ops.reduce_max(singular_values, axis=-1)\n", - " singular_values = array_ops.where_v2(\n", - " singular_values > array_ops.expand_dims_v2(cutoff, -1), singular_values**beta,\n", - " np.array(np.inf, dtype))\n", - "\n", - " # By the definition of the SVD, `a == u @ s @ v^H`, and the pseudo-inverse\n", - " # is defined as `pinv(a) == v @ inv(s) @ u^H`.\n", - " a_pinv = math_ops.matmul(\n", - " right_singular_vectors / array_ops.expand_dims_v2(singular_values, -2),\n", - " left_singular_vectors,\n", - " adjoint_b=True)\n", - "\n", - " if a.shape is not None and a.shape.rank is not None:\n", - " a_pinv.set_shape(a.shape[:-2].concatenate([a.shape[-1], a.shape[-2]]))\n", - "\n", - " return a_pinv" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We define a Python class to organize the optimization algorithm. Using physics solver, network model, loss function and a data set. We compute loss values after each epoch. Each epoch consists of several mini batch updates. Depending on the optimization method, the mini batch update differs:\n", - "1. Adam: Compute loss gradient, then apply the Adam\n", - "2. PG: Compute loss gradient und physics Jacobian, invert them data-point-wise, define a modified loss to compute network updates\n", - "3. HIG: Compute loss gradient and network-physics Jacobian, then compute half-inversion" - ] - }, - { - "cell_type": "code", - "execution_count": 86, - "metadata": {}, - "outputs": [], - "source": [ - "class Optimization():\n", - " def __init__(self,model,solver,loss_function,x_train,y_train):\n", - " self.model = model\n", - " self.solver = solver\n", - " self.loss_function = loss_function\n", - " self.x_train = x_train\n", - " self.y_train = y_train\n", - " self.y_dim = y_train.shape[1]\n", - " self.weight_shapes = [weight_tensor.shape for weight_tensor in self.model.trainable_weights]\n", - "\n", - " def set_params(self,batch_size,learning_rate,optimizer,max_number,mode,trunc):\n", - " self.number_of_batches = N // batch_size\n", - " self.max_number = max_number\n", - " self.batch_size = batch_size\n", - " self.learning_rate = learning_rate\n", - " self.optimizer = optimizer(learning_rate)\n", - " self.mode = mode\n", - " self.trunc = trunc\n", - "\n", - "\n", - " def computation(self,x_batch, y_batch):\n", - " control_batch = self.model(y_batch)\n", - " y_prediction_batch = self.solver(x_batch,control_batch)\n", - " loss = self.loss_function(y_batch,y_prediction_batch)\n", - " return loss\n", - "\n", - "\n", - " @tf.function\n", - " def gd_get_derivatives(self,x_batch, y_batch):\n", - "\n", - " with tf.GradientTape(persistent=True) as tape:\n", - " tape.watch(self.model.trainable_variables)\n", - " loss = self.computation(x_batch,y_batch)\n", - " loss_per_dp = loss / self.batch_size\n", - " grad = tape.gradient(loss_per_dp, self.model.trainable_variables)\n", - " return grad\n", - "\n", - "\n", - " @tf.function\n", - " def pg_get_physics_derivatives(self,x_batch, y_batch): #physical grads\n", - "\n", - " with tf.GradientTape(persistent=True) as tape:\n", - " control_batch = self.model(y_batch)\n", - " tape.watch(control_batch)\n", - " y_prediction_batch = self.solver(x_batch,control_batch)\n", - " loss = self.loss_function(y_batch,y_prediction_batch)\n", - " loss_per_dp = loss / self.batch_size\n", - "\n", - " jacy = tape.batch_jacobian(y_prediction_batch,control_batch)\n", - " grad = tape.gradient(loss_per_dp, y_prediction_batch)\n", - " return jacy,grad,control_batch\n", - "\n", - " @tf.function\n", - " def pg_get_network_derivatives(self,x_batch, y_batch,new_control_batch): #physical grads\n", - "\n", - " with tf.GradientTape(persistent=True) as tape:\n", - " tape.watch(self.model.trainable_variables)\n", - " control_batch = self.model(y_batch)\n", - " loss = self.loss_function(new_control_batch,control_batch)\n", - " #y_prediction_batch = self.solver(x_batch,control_batch)\n", - " #loss = self.loss_function(y_batch,y_prediction_batch)\n", - " loss_per_dp = loss / self.batch_size\n", - "\n", - "\n", - "\n", - " network_grad = tape.gradient(loss_per_dp, self.model.trainable_variables)\n", - " return network_grad\n", - "\n", - " @tf.function\n", - " def hig_get_derivatives(self,x_batch,y_batch):\n", - "\n", - " with tf.GradientTape(persistent=True) as tape:\n", - " tape.watch(self.model.trainable_variables)\n", - " control_batch = self.model(y_batch)\n", - " y_prediction_batch = self.solver(x_batch,control_batch)\n", - " loss = self.loss_function(y_batch,y_prediction_batch)\n", - " loss_per_dp = loss / self.batch_size\n", - "\n", - " jacy = tape.jacobian(y_prediction_batch, self.model.trainable_variables, experimental_use_pfor=True)\n", - " loss_grad = tape.gradient(loss_per_dp, y_prediction_batch)\n", - " return jacy, loss_grad\n", - "\n", - "\n", - "\n", - " def mini_batch_update(self,x_batch, y_batch):\n", - " if self.mode==\"GD\":\n", - " grad = self.gd_get_derivatives(x_batch, y_batch)\n", - " self.optimizer.apply_gradients(zip(grad, self.model.trainable_weights))\n", - "\n", - " elif self.mode==\"PG\":\n", - " jacy,grad,control_batch = self.pg_get_physics_derivatives(x_batch, y_batch)\n", - " grad_e = tf.expand_dims(grad,-1)\n", - " pinv = tf.linalg.pinv(jacy,rcond=10**-5)\n", - " delta_control_label_batch = (pinv@grad_e)[:,:,0]\n", - " new_control_batch = control_batch - delta_control_label_batch\n", - " network_grad = self.pg_get_network_derivatives(x_batch, y_batch,new_control_batch)\n", - " self.optimizer.apply_gradients(zip(network_grad, self.model.trainable_weights))\n", - "\n", - " elif self.mode =='HIG':\n", - " jacy, grad = self.hig_get_derivatives(x_batch, y_batch)\n", - " flat_jacy_list = [tf.reshape(jac, (self.batch_size * self.y_dim, -1)) for jac in jacy]\n", - " flat_jacy = tf.concat(flat_jacy_list, axis=1)\n", - " flat_grad = tf.reshape(grad, (-1,))\n", - " inv = gpinv(flat_jacy, rcond=self.trunc)\n", - " processed_derivatives = tf.tensordot(inv, flat_grad, axes=(1, 0))\n", - " #processed_derivatives = self.linear_solve(flat_jacy, flat_grad)\n", - " update_list = []\n", - " l1 = 0\n", - " for k, shape in enumerate(self.weight_shapes):\n", - " l2 = l1 + np.prod(shape)\n", - " upd = processed_derivatives[l1:l2]\n", - " upd = np.reshape(upd, shape)\n", - " update_list.append(upd)\n", - " l1 = l2\n", - " self.optimizer.apply_gradients(zip(update_list, self.model.trainable_weights))\n", - "\n", - "\n", - " def epoch_update(self):\n", - " for batch_index in range(self.number_of_batches):\n", - " position = batch_index * batch_size\n", - " x_batch = self.x_train[position:position + batch_size]\n", - " y_batch = self.y_train[position:position + batch_size]\n", - " self.mini_batch_update(x_batch, y_batch)\n", - "\n", - " def eval(self,epoch,wc_time,ep_dur):\n", - "\n", - " print('Epoch: ', epoch,' WallClockTime: ',wc_time,' EpochDuration: ',ep_dur )\n", - "\n", - " train_loss = self.computation(self.x_train,self.y_train)\n", - " train_loss_per_dp = train_loss / N\n", - " print('TrainLoss:', train_loss_per_dp)\n", - " return train_loss_per_dp\n", - "\n", - " def start_training(self):\n", - " init_loss = self.eval(0,0,0)\n", - " init_time = time.time()\n", - " time_list = [init_time]\n", - " loss_list = [init_loss]\n", - "\n", - " epoch=0\n", - " wc_time = 0\n", - "\n", - " while wc_time" + "cell_type": "markdown", + "source": [ + "## Problem Statement\n", + "\n", + "We consider a two-dimensional system governed by the heat equation $\\frac{\\partial u}{\\partial t} = \\nu \\cdot \\nabla^2 u$.\n", + "Given an initial state $x = u(t_0)$ at $t_0$, the simulator computes the state at a later time $t_*$ via $y = u(t_*) = \\mathcal P(x)$.\n", + "Exactly inverting this system is only possible for $t \\cdot \\nu = 0$ and becomes increasingly unstable for larger $t \\cdot \\nu$ because initially distinct heat levels even out over time, drowning the original information in noise.\n", + "Hence the Jacobian of the physics $\\frac{\\partial y}{\\partial x}$ is near-singular.\n", + "\n", + "We'll use periodic boundary conditions and compute the result in frequency space where the physics can be computed analytically as $\\hat y = \\hat x \\cdot e^{-k^2 (t_* - t_0)} $ , where $\\hat y_k \\equiv \\mathcal F(y)_k$ denotes the $k$-th element of the Fourier-transformed vector $y$.\n", + "In the regular _forward_ simulation, high frequencies are dampened exponentially. We'll need to revisit this aspect for the inverse simulator.\n", + "\n", + "To summarize, the inverse problem we're targeting here can be written as minimizing: \n", + "\n", + "$$L(x) = || \\mathcal P(x) - y^* ||_2^2 = || \\mathcal F^{-1}\\left( \\mathcal F(x) \\cdot e^{-k^2 (t_* - t_0)} \\right) - y^* ||_2^2 . $$\n", + "\n" + ], + "metadata": { + "id": "9fETS9hwrQrc" + } + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## Implementation\n", + "\n", + "Below, we'll set $t\\cdot \\nu = 8$ on a domain consisting of 64x64 cells of unit length.\n", + "This level of diffusion is challenging, and diffuses most details while leaving only the large-scale structure intact.\n", + "\n", + "We'll use phiflow with PyTorch as default backend, but this example code likewise runs with TensorFlow (just switch to `phi.tf.flow` below).\n", + "\n" + ], + "metadata": { + "id": "EA7qc5mXsZLU" + } + }, + { + "cell_type": "code", + "execution_count": 143, + "outputs": [], + "source": [ + "!pip install --upgrade --quiet git+https://github.com/tum-pbs/PhiFlow\n", + "from phi.torch.flow import * # switch to TF with \"phi.tf.flow\"" + ], + "metadata": { + "pycharm": { + "name": "#%%\n" + }, + "id": "vN06frDqS2eM" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "## Data generation\n", + "\n", + "For training, we generate $x^*$ by randomly placing between 4 and 10 \"hot\" rectangles of random size and shape in the domain. The `generate_heat_example()` function below generates a full mini batch (via `shape.batch`) of example positions. These will be $x$ later on. They're never passed to the solver, but should be reconstructed by the neural network.\n", + "\n" + ], + "metadata": { + "collapsed": false, + "id": "fc6Am-XcS2eN" + } + }, + { + "cell_type": "code", + "execution_count": 144, + "outputs": [], + "source": [ + "def generate_heat_example(*shape, bounds: Box = None):\n", + " shape = math.merge_shapes(*shape)\n", + " heat_t0 = CenteredGrid(0, extrapolation.PERIODIC, bounds, resolution=shape.spatial)\n", + " bounds = heat_t0.bounds\n", + " component_counts = math.to_int32(4 + 7 * math.random_uniform(shape.batch))\n", + " positions = (math.random_uniform(shape.batch, batch(components=10), channel(vector=shape.spatial.names)) - 0.5) * bounds.size * 0.8 + bounds.size * 0.5\n", + " for i in range(10):\n", + " position = positions.components[i]\n", + " half_size = math.random_uniform(shape.batch, channel(vector=shape.spatial.names)) * 10\n", + " strength = math.random_uniform(shape.batch) * math.to_float(i < component_counts)\n", + " position = math.clip(position, bounds.lower + half_size, bounds.upper - half_size)\n", + " component_box = Cuboid(position, half_size)\n", + " component_mask = SoftGeometryMask(component_box)\n", + " component_mask = component_mask.at(heat_t0)\n", + " heat_t0 += component_mask * strength\n", + " return heat_t0" + ], + "metadata": { + "pycharm": { + "name": "#%%\n" + }, + "id": "-39wQc1ES2eN" + } + }, + { + "cell_type": "markdown", + "source": [ + "The data is generated on-the-fly later during training, but let's look at two example $x$ for now using phiflow's `vis.plot` function:" + ], + "metadata": { + "collapsed": false, + "id": "sz_eZyZ7S2eO" + } + }, + { + "cell_type": "code", + "execution_count": 145, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAz4AAAFgCAYAAAB60Kx2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df7BkdXnn8fczM8DID+XHxAlhMExWjGGjgDshEq0NSsyOrgXWxqJkEzOmyE5lK6Z0k2zEbFZ3SbIbk90YU0WymVUiSamEJTFMJSRIENckKjIERJgRHVHCIDiCYFiNDHPvs3/0GXOZOz/udPe3Tz9936+qrtvn2327n4au+5nnnO/5nshMJEmSJGmWrei7AEmSJElqzcZHkiRJ0syz8ZEkSZI082x8JEmSJM08Gx9JkiRJM29V3wUsxdFxTK7muL7LkKSRfJOvsyefjHG+5r962XH56Ffnhvrd2+968sbM3DjOemaN+SNpFow7f0bJHugvf0o0Pqs5ju+PC/suQ5JGcmvePPbXfPSrc3zyxucM9bsrT/3cmjGXM3PMH0mzYNz5M0r2QH/5U6LxkSQdWALzzPddhiRpGamaPTY+klRaMpf1wkeSVFnN7HFxA0mSJEkzzyM+klTYYLpB9l2GJGkZqZo9Nj6SVFzFedaSpNoqZo+NjyQVliRzWW+vmySprqrZY+MjScVVnG4gSaqtYvbY+EhSYQnMFQwfSVJdVbPHVd0kSQcUEVdFxO6IuPsgj/9oRNwVEZ+OiI9FxNmTrlGSNHta5Y+NjyQVN08OdVuC9wIbD/H4F4AfzMwXAL8MbBn900iSKhg2e/rMH6e6SVJhCc1OMM3Mj0bEGYd4/GMLNj8BrGtSiCRpqrTMHmiXPzY+klTcCAuKromIbQu2t2TmsEdtLgP+YvhSJEmVjLiYdS/5Y+MjSYUlOcoJpo9k5oZRa4iIlzEInpeO+lqSpOk3YvZAT/lj4yNJlSXM9biwTkS8EHg38MrMfLS/SiRJE9Nz9sBw+dN0cYOIODEirouIz0TEjog4PyJOjoibIuJz3c+TWtYgSWojIp4D/Anw+sz8bN/1LGT+SNLsGjZ/Wq/q9i7gLzPz+cDZwA7gcuDmzDwTuLnbliQNIRnMsx7mdjgR8QHg48B3R8SuiLgsIn4qIn6qe8rbgFOA34mIO/ebr90380eSGhkle/rMn2ZT3SLiWcC/BN4AkJl7gD0RcTFwQfe0q4GPAG9pVYckzbZgjmjyypl56WEe/0ngJ5u8+QjMH0lqrV32QLv8aXnEZz3wFeD3I+KOiHh3RBwHrM3Mh7rnPAysbViDJM20BOZzuNsMM38kqaFRsqfP/GnZ+KwCXgT8bmaeC3yd/aYVZGbCgZeEiIjNEbEtIrY9xZMNy5Sk2ua6PW9Hepth5o8kNTZs9vSZPy0bn13Arsy8tdu+jkEQfTkiTgXofu4+0C9n5pbM3JCZG47imIZlSlJdiY3PAZg/ktTQKNkzk41PZj4MPBAR390NXQhsB7YCm7qxTcD1rWqQJC0/5o8k6UBaX8fnZ4D3RcTRwH3ATzBotq6NiMuA+4FLGtcgSTNtPmf66M2wzB9Jaqhi9jRtfDLzTuBAV2W9sOX7StJysW+6gZ7O/JGkdqpmT+sjPpKkhpJgrvkl2SRJ+idVs8fGR5KKqzjdQJJUW8XssfGRpMKqTjeQJNVVNXvqHaOSJEmSpCPkER9JKi2YS/dhSZImqWb22PhIUmEJzHvwXpI0QVWzx8ZHkoqrOM9aklRbxeyx8ZGkwjJrTjeQJNVVNXtsfCSpuPmCe90kSbVVzB4bHw0n6n3Zxyaz7wokSZJ0hGx8JKmwwbUU6k03kCTVVTV7bHwkqbSa86wlSZXVzB4bH0kqrOqSopKkuqpmj42PJBU3l8v4nDtJUi8qZo+NjyQVlkTJedaqZ8UJJywai9Wrh369ua98ZZRyJPWoavbUq1iSJEmSjpBHfCSpuPmCJ5hKkmqrmD02PpJUWNUlRSVJdVXNHhsfSSosiZInmGo2PPnC5ywamztm8T+GnvGlry/+Zc/xkcqqmj02PpJUXMUlRSVJtVXMHhsfSSosk5IXkZMk1VU1e+pVLEmSJElHyCM+klRaME+9edaSpMpqZo+NjyQVltScbqDZlSvq/WNI0pGpmj02PpJUXMUlRSVJtVXMHhsfSSosCeYLLikqSaqravbY+EhScRX3ukmSaquYPfUqliRJkqQj5BEfSSosgfmCJ5hKkuqqmj02PpJUWjBXcElRSVJlNbPHxkeSCqu6102SVFfV7LHxkaTiKu51kyTVVjF7bHwkqbDMKLnXTZJUV9XssfHRYT3++vP7LgGAE//w432XIEmSpKLqtWqSpKeZyxVD3Q4nIq6KiN0RcfdBHo+I+O2I2BkRd0XEi8b+4SRJU2nY7Okzf2x8JKmwBOaJoW5L8F5g4yEefyVwZnfbDPzuqJ9HkjT9RsmePvPHqW6SVFosae/ZMDLzoxFxxiGecjHwB5mZwCci4sSIODUzH2pSkCRpSrTLHmiXPzY+klTYYEnRoVfWWRMR2xZsb8nMLUfw+6cBDyzY3tWN2fhI0gwbMXugp/yx8dFh7Xnm4i/2/Csea/qeq/7ixKavL82SueFnLT+SmRvGWYskaXkYIXugp/xp2vhExBeBJ4A5YG9mboiIk4E/As4Avghckplt/xUtSWrhQeD0BdvrurHemT+SNNOGyp9JLG7wssw8Z0FXdzlwc2aeCdzcbUuShpAE8zncbQy2Aj/era7zYuBrU3Z+j/kjSQ2Mkj195k8fU90uBi7o7l8NfAR4Sw91SNJMmG+0DysiPsDg7/WaiNgFvB04CiAz/xdwA/AqYCfwDeAnmhQyPuaPJI1Jq+yBdvnTuvFJ4EMRkcDvdSctrV3QkT0MrG1cgyTNrEyYG8/eswO8dl56mMcT+Okmbz4680eSGmmZPYPXb5M/rRufl2bmgxHxbOCmiPjMwgczM7tQWiQiNjNYl5vVHNu4TEmqa0zTBmaN+SNJDVXMnqaNT2Y+2P3cHREfBM4Dvrxvne2IOBXYfZDf3QJsAXhmnHzAcJKk5W4wz9prUe/P/JGkdqpmT7OKI+K4iDhh333gh4G7GZyMtKl72ibg+lY1SJKWH/NHknQgLY/4rAU+GBH73uf9mfmXEXEbcG1EXAbcD1zSsAZJmnlz1Jtu0Jj5I0mNVcyeZo1PZt4HnH2A8UeBC1u9ryQtJ2O4evbMMX8kqa2q2dPHctaSpLGpOc9aklRZzeyx8ZGk4uYLTjeQJNVWMXtsfCSpsNbXUpAkaX9Vs6feMSpJkiRJOkIe8ZGk4irOs5Yk1VYxe2x8JKmwwUXk6k03kCTVVTV7bHwkqbiKJ5hKkmqrmD02PpJUWNVrKUiS6qqaPTY+klRcxXnWkqTaKmZPvYolSZIk6Qh5xEeSKsuaJ5hKkgormj02PpJUWFLzBFNJUl1Vs8fGR5KKq7jXTZJUW8XssfGRpMKqrqwjSaqravbY+EhScRXDR5JUW8XscVU3SZIkSTPPIz6SVFhSc2UdSVJdVbPHxkeSiqu4so4kqbaK2WPjI0mVZc151pKkwopmj42PJBVWdWUdSVJdVbPHxkeSiqsYPpKk2ipmj6u6SZIkSZp5HvGRpMKqrqwjSaqravbY+EhScVkwfCRJtVXMHhsfSSqu4pKikqTaKmaPjY8kFZZFlxSVJNVVNXtsfCSpuIrTDSRJtVXMHld1kyRJkjTzyh7xWfnc9YsHV8xeHzd/3/1P2869e3uqRNJ0qrmyjiSpsprZU7bxkSQNVJxuIEmqrWL22PhIUmFJzRNMJUl1Vc0eGx9JqiwHq+tIkjQxRbPHxkeSiqt4LQVJUm0Vs2emGp/HNnxb3yWMZPWjixcuOGa/xQ0kSZIkHbmZanwkablJap5gKkmqq2r2zN76z5K0rAyWFB3mtqRXj9gYEfdGxM6IuPwAjz8nIm6JiDsi4q6IeNXYP6IkacoMnz1LyZ9W2WPjI0nFZQ53O5yIWAlcCbwSOAu4NCLO2u9pvwRcm5nnAq8Dfme8n06SNI2GzZ7D5U/L7HGqmyQV13C6wXnAzsy8DyAirgEuBrYvfHvgmd39ZwFfalWMJGl6VMweGx9JKmyw92zo8FkTEdsWbG/JzC0Ltk8DHliwvQv4/v1e478AH4qInwGOA35o2GJUz+rtDw79u/NjrEPSZI2YPXDo/GmWPc0bn+5w1Tbgwcx8dUSsB64BTgFuB16fmXta1yFJWuSRzNww4mtcCrw3M/9nRJwP/GFEfG9m9v7vWvNHkqbWqPkzVPZM4hyfNwE7Fmy/A3hnZj4XeAy4bAI1SNLMari4wYPA6Qu213VjC10GXAuQmR8HVgNrxvCxxsH8kaRGGi5u0Cx7mjY+EbEO+NfAu7vtAF4OXNc95WrgNS1rkKRZ12pxA+A24MyIWB8RRzM4gXTrfs/5e+BCgIj4Hgbh85XxfbrhmD+S1FarxQ1omD2tp7r9FvALwAnd9inA45m570qduxjM45MkDanVCaaZuTci3gjcCKwErsrMeyLiCmBbZm4Ffg743xHxHxicbPqGzCW2VW2ZP5LUUMXsadb4RMSrgd2ZeXtEXDDE728GNgOs5tgxV6dRrbjppMbvMA3/bpKmXxJNLyKXmTcAN+w39rYF97cDL2lWwBDMnzbmn3hiSWPLxfxLz+m7hCVb8Td39l2CZkzV7Gl5xOclwEXdBYVWM1hy7l3AiRGxqtvrdqA5ewB0KztsAXhmnOy/giXpIPwDuYj5I0mNVfzj2Owcn8x8a2auy8wzGMzN+3Bm/ihwC/Da7mmbgOtb1SBJWn7MH0nSgUxiVbf9vQX42YjYyWDO9Xt6qEGSZkN3LYVhbsuQ+SNJ4zBC9vSZPxO5gGlmfgT4SHf/PgZXZFURz77yY32XIOlQKs43mBDzR63kqsX7jj+/qf8dCuvft3isj73cWgYKZs9EGh9JUjvL9OiNJKlHFbPHxkeSipuKxaMlSctKxeyx8ZGkwpKae90kSXVVzR6nfUqSJEmaeR7xkaTKEii4102SVFjR7LHxkaTiKs6zliTVVjF7bHwkqbqC4SNJKq5g9tj4SFJpy/ZipJKk3tTMHhsfSaqu4F43SVJxBbPHVd0kSZIkzTyP+EhSZVnzWgqSpMKKZo+NjyRVV3C6gSSpuILZY+MjSeXV2+smSaquXvbY+EhSdQX3ukmSiiuYPTY+klRdwfCRJBVXMHtc1U2SJEnSzPOIjyRVlkDBlXUkSYUVzR4bH0kqLgtON5Ak1VYxe2x8JKm6guEjSSquYPbY+EhSdQWnG0iqacWxxy4am//GN3qoRL0rmD02PpJUXBTc6yZJqq1i9riqmyRJkqSZ5xEfSaosKTnPWpJUWNHssfGRpNKi5DxrSUU99zmLx+76zOTrUM9qZo+NjyRVV3CvmySpuILZY+MjSdUVDB9JUnEFs8fGR5KqKxg+kqTiCmaPq7pJkiRJmnke8ZGkypKSJ5hKkgormj02PpJUXMWLyEmSaquYPTY+klRdwfCRJBVXMHs8x0eSJEnSzDts4xMRPxMRJ02iGEnSkYsc7jbtzB9Jml7DZk+f+bOUIz5rgdsi4tqI2BgR9c5kkiRVZP5IksbmsI1PZv4ScCbwHuANwOci4r9FxD9rXJskaSkyhrtNOfNHkqbYsNnTY/4s6RyfzEzg4e62FzgJuC4ifr1hbZKkw8kRbgWYP5I0hUbJnmme6hYRb4qI24FfB/4WeEFm/nvgXwA/0rg+SdLhNAyeborZvRGxMyIuP8hzLomI7RFxT0S8f8RPs/B1zR9JmlYNG59W2bOU5axPBv5NZt6/cDAz5yPi1Ut5E0lSO61OFI2IlcCVwCuAXQzOt9mamdsXPOdM4K3ASzLzsYh49hhLMH8kaUpVzJ7DNj6Z+fZDPLZjKW8iSWqo3bSB84CdmXkfQERcA1wMbF/wnH8HXJmZjwFk5u5xvbn5I0lTrGD2eB0fSVq+1kTEtgW3zfs9fhrwwILtXd3YQs8DnhcRfxsRn4iIjS0LliTNhEPlT7PsWcpUt6FExGrgo8Ax3ftcl5lvj4j1wDXAKcDtwOszc0+rOiRp5g2/1+2RzNww4ruvYrDy2gXAOuCjEfGCzHx8xNcdmvkjSRMw2hGfUfNnqOxpecTnSeDlmXk2cA6wMSJeDLwDeGdmPhd4DLisYQ2SNNMaX0DuQeD0BdvrurGFdgFbM/OpzPwC8FkGYdQn80eSGhole5aQP82yp1njkwP/r9s8qrsl8HLgum78auA1rWqQpGWh3XUUbgPOjIj1EXE08Dpg637P+VMGe9yIiDUMph/cN74Pd+TMH0magHbX8WmWPU3P8YmIlRFxJ7AbuAn4PPB4Zu7tnnKgOXv7fnfzvnl/T/FkyzIlqbZGy4l2f6vfCNwI7ACuzcx7IuKKiLioe9qNwKMRsR24BfiPmfno2D7bkMwfSWqs0XLWLbOn2Tk+AJk5B5wTEScCHwSefwS/uwXYAvDMOLnIpfYkafJaLSkKkJk3ADfsN/a2BfcT+NnuNjXMH0lqq2L2TGRVt+5Eo1uA84ETI2Jfw3WgOXuSJI2F+SNJ2qdZ4xMR39btaSMinsHgIkQ7GATQa7unbQKub1WDJC0Ljaa6VWX+SNIENJrq1lLLqW6nAld3V19dwWB+3p91c/GuiYhfAe4A3tOwBkmabUtfoW05MX8kqaWi2dOs8cnMu4BzDzB+H4MrskqSxqFg+LRk/kjSBBTMnqaLG0iSJqBg+EiSiiuYPTY+klRcxekGkqTaKmbPRFZ1kyRJkqQ+ecRHkiSVser0dX2XAMATa49eNHbSKY/1UMnTPXnSiYvGjuqhDmka2fhIUnUFpxtIkoormD02PpJUWdElRSVJhRXNHhsfSaquYPhIkoormD02PpJUXcHwkSQVVzB7bHwkSVIZe75zzaKxh1987MTr+MbZ/7ho7P+e856J1vC1+ZWLxjbz5onWIFVi4yNJhQU151lLkuqqmj02PpJUXcHwkSQVVzB7bHwkqbKiK+tIkgormj02PpJUXcHwkSQVVzB7Zqrxibm+K5CkHhQMH2mc9k5+bQNWH7tn0di6VcdPtIav7Vm8wEJrj73wxEVjz1r9grZv+slPt319Dadg9qzouwBJkiRJam2mjvhI0nJUcZ61JKm2itlj4yNJ1RUMH0lScQWzx8ZHkipLSoaPJKmwotlTtvGJf3xy0dhJf/P3PVTSlus1SDqcitMNJNW05/hYNPb5N68c2+uv+fPVi8ae9cmxvbzGqGL2lG18JEmdguEjSSquYPa4qpskSZKkmecRH0kqruJ0A0lSbRWzx8ZHkqorGD6SpOIKZk/Zxmfvg1/quwSpNytOOKHvEmbC/BNP9F3C6IqurCNpdqxYMd93CZq0otlTtvGRJEF0N0mSJqVq9tj4SFJ1Bfe6SZKKK5g9ruomSZIkaeZ5xEcq6OEff8GiscfPeaqHSuo4+uHFf+7O+M8f76GS8au4so40i97+lX8+0ffb8cS3T/T9AFb94+KxM357nJOeDvAGmkoVs8fGR5KqKxg+kqTiCmaPjY8kVVcwfCRJxRXMHhsfSaosa043kCQVVjR7bHwkqbqC4SNJKq5g9sx043P/FecvGps/uodCDvDFWP/W2Tipehy+9PM/0HcJvTnttz65aCz37h3qtb7ve+4btZyZ9qmHn9d3CZJmyPynnrVo7NpP/eBEazjm8cVj33HLvYvG5sb4nqfc8diisfm7PjPGd5DamenGR5KWg4rTDSRJtVXMHhsfSaquYPhIkoormD02PpJUXMW9bpKk2ipmz4q+C5AkjSBHuC1BRGyMiHsjYmdEXH6I5/1IRGREbBj+w0iSShgle5aQP62yZ9kd8XnVK26b+Hve8KHvm/h7VvId/+NjfZfQm4I7SzSNGn2RImIlcCXwCmAXcFtEbM3M7fs97wTgTcCtbSqRJE2dgtnjER9J0sGcB+zMzPsycw9wDXDxAZ73y8A7gG9OsjhJ0kxqlj3NGp+IOD0ibomI7RFxT0S8qRs/OSJuiojPdT9PalWDJM26YDDPepgbsCYiti24bd7v5U8DHliwvasb+6f3j3gRcHpm/nnDj3lEzB9JamuU7FlC/jTLnpZT3fYCP5eZf9cdiro9Im4C3gDcnJm/1s3Zuxx4S8M6JGm2DT/d4JHMHPqcnIhYAfwmg7/r08T8kaTWRpvqNnT+jJI9zY74ZOZDmfl33f0ngB0MurWLgau7p10NvKZVDZK0HETmULcleBA4fcH2um5snxOA7wU+EhFfBF4MbO17gQPzR5LaGzZ7lpA/zbJnIosbRMQZwLkMTj5am5kPdQ89DKw9yO9sBjYDrObY9kVKUkVHsELbEG4DzoyI9QxC53XAv/3WW2d+DVizbzsiPgL8fGZua1bRETJ/JKmBotnTfHGDiDge+GPgzZn5Dwsfy8yD/mfLzC2ZuSEzNxzFMa3LlKSyRphjfUiZuRd4I3Ajg6Mm12bmPRFxRURc1PZTjc78kaR2RjzH56BaZk/TIz4RcRSD0HlfZv5JN/zliDg1Mx+KiFOB3S1rkCQNLzNvAG7Yb+xtB3nuBZOoaSnMH0mqq1X2tFzVLYD3ADsy8zcXPLQV2NTd3wRc36oGSVoWGl7AtCLzR5ImoPEFtFtoecTnJcDrgU9HxJ3d2C8CvwZcGxGXAfcDlzSsQZJm3lKmrS0z5o8kNVYxe5o1Ppn5NwyW+T6QC1u9r6bbitWrF439w0Xn9FBJHSv2FvzLosnyK/I05o8kTUDB7JnIqm6SpEaWuFCBJEljUzR7bHwkqbqC4SNJKq5g9jRfzlqSJEmS+uYRH/XuqWMXT8X/xrcfbHr+8nPSZ+f6LkFTLKg53UBSTfHNp/ouQVOgavbY+EhSdVkwfSRJtRXMHhsfSSqu4l43SVJtFbPHxkeSKpvxi5FKkqZQ0eyx8ZGk4mK+7wokSctNxeyx8ZEkSaUdv6vgruei5j77+b5LkIZm4yNJ1flvPknSpBXMHhsfSSqu4gmmkqTaKmaPjY8kVZaUXFJUklRY0eyx8ZGk4irudZMk1VYxe2x8JKm6guEjDWvlJ+5eNHbKJ3ooZEoVXGhLVRXMnhV9FyBJkiRJrXnER5IKC2pON5Ak1VU1e2x8JKmyzJInmEqSCiuaPTY+klRcxb1ukqTaKmaPjY8kVVcwfKRh5d69fZcgCUpmj42PJBVXca+bJKm2itnjqm6SJEmSZp5HfCSpsgTmC+52kyTVVTR7bHwkqbp62SNJqq5g9tj4SDPijr99Xt8lTLVZntdbcZ61JKm2itlj4yNJ1RW8loIkqbiC2WPjI0nFVdzrJkmqrWL2zPLsD0mSJEkCPOIjlfTsKz+2eKyHOjQFkpInmEqSCiuaPTY+klRYAFFwnrUkqa6q2WPjI0nVzfddgCRp2SmYPTY+klRcxb1ukqTaKmaPjY8kVVZ0nrUkqbCi2eOqbpIkSZJmnkd8JKm0LHkROUlSZTWzx8ZHkoqreBE5SVJtFbPHxkeSqiu4102SVFzB7LHxkaTKEqLgkqKSpMKKZo+LG0hSdZnD3ZYgIjZGxL0RsTMiLj/A4z8bEdsj4q6IuDkivnPsn0+SNH2GzZ4l5E+r7LHxkSQdUESsBK4EXgmcBVwaEWft97Q7gA2Z+ULgOuDXJ1ulJGmWtMyeZo1PRFwVEbsj4u4FYydHxE0R8bnu50mt3l+Slo0c8nZ45wE7M/O+zNwDXANc/LS3zrwlM7/RbX4CWDfipxmZ+SNJEzBs9hw+f5plT8sjPu8FNu43djlwc2aeCdzcbUuSRhCZQ92W4DTggQXbu7qxg7kM+IsRPsq4vBfzR5KaGjZ7lpA/zbKnWeOTmR8Fvrrf8MXA1d39q4HXtHp/SVo2hp9jvSYiti24bR62hIj4MWAD8Bvj+ljDMn8kaQJGO8dnLPlzpNkz6VXd1mbmQ939h4G1B3ti9x9gM8Bqjp1AaZJUUALDr6zzSGZuOMTjDwKnL9he1409TUT8EPCfgB/MzCeHrqYt80eSxmW07IFD50+z7OltcYPMPOQsv8zckpkbMnPDURwzwcokqY6g2VQDgNuAMyNifUQcDbwO2Pq09484F/g94KLM3D32D9iA+SNJoxkle5aQP82yZ9KNz5cj4lSA7meJkJSk5Sgz9wJvBG4EdgDXZuY9EXFFRFzUPe03gOOB/xMRd0bE1oO8XN/MH0kqoGX2THqq21ZgE/Br3c/rJ/z+kjR7Gl49OzNvAG7Yb+xtC+7/ULM3Hy/zR5LGqWD2NGt8IuIDwAUMTl7aBbydQeBcGxGXAfcDl7R6f0laNhqGT0XmjyRNQMHsadb4ZOalB3nowlbvKUnLzugnmM4c80eSGiuaPZOe6iZJGrMlLlQgSdLYVMweGx9Jqq5g+EiSiiuYPb0tZy1JkiRJk+IRH0kqLUvudZMkVVYze2x8JKmypGT4SJIKK5o9Nj6SVF3BlXUkScUVzB4bH0kqruLKOpKk2ipmj42PJFVXMHwkScUVzB5XdZMkSZI08zziI0mVJTBfb6+bJKmwotlj4yNJpdVcUlSSVFnN7LHxkaTqCoaPJKm4gtlj4yNJ1RUMH0lScQWzx8ZHkiorOs9aklRY0exxVTdJkiRJM88jPpJUWkIWvHy2JKmwmtlj4yNJ1RWcZy1JKq5g9tj4SFJlRedZS5IKK5o9y67x+esvfVffJUjSeBXc6yZJKq5g9iy7xkeSZk7B8JEkFVcwe1zVTZIkSdLM84iPJJWWJfe6SZIqq5k9Nj6SVFkC8/WWFJUkFVY0e2a68fmuP3h48eAfTL6OtSyuY27yZUiaVQX3ukmSiiuYPTPd+EjSslAwfCRJxRXMHhsfSSotS15LQZJUWc3scVU3SZIkSTPPIz6SVFlCZr0TTCVJhRXNnplufOZ2fqHvEiSpvYLTDSRJxRXMnplufCRpWSh4gqkkqbiC2WPjI0mVZZa8loIkqbCi2WPjI0nVFdzrJkkqrmD22Piod2s++WjfJZTjBXAlSZKOjI2PJBWXBacbSJJqq5g9Nj6SVFqWnG4gSaqsZvbY+EhSZUnJJUUlSYUVzR4bH0mqruBF5CRJxRXMHhsfTdT8N7+5eHD7Z2CYIpYAAAc2SURBVCdfiDQjEsiCe90kSXVVzZ4VfRcgSZIkSa310vhExMaIuDcidkbE5X3UIEkzIXMw3WCY2xIc7u91RBwTEX/UPX5rRJwx5k84VuaPJI3BKNmzhPxplT0Tb3wiYiVwJfBK4Czg0og4a9J1SNKsyPkc6nY4S/x7fRnwWGY+F3gn8I4xf7yxMX8kaXyGzZ7D5U/L7OnjiM95wM7MvC8z9wDXABf3UIckzYZ2R3yW8vf6YuDq7v51wIUREWP7bONl/kjSuLQ74tMse/pY3OA04IEF27uA79//SRGxGdjcbT75V3nd3ROorZU1wCN9FzEC6++X9fdrnPV/55he51ue4LEb/yqvWzPkr6+OiG0Ltrdk5pYF20v5e/2t52Tm3oj4GnAK0/n/3Pypx/r7Zf39mtr8GTF74ND50yx7pnZVt+7DbwGIiG2ZuaHnkoZm/f2y/n5Zf1uZubHvGmaN+TM9rL9f1t+vaa6/avb0MdXtQeD0BdvrujFJ0nRZyt/rbz0nIlYBzwIenUh1R878kaTp1yx7+mh8bgPOjIj1EXE08Dpgaw91SJIObSl/r7cCm7r7rwU+nJnTenEH80eSpl+z7Jn4VLduHt4bgRuBlcBVmXnPYX5ty2Een3bW3y/r75f1F3Wwv9cRcQWwLTO3Au8B/jAidgJfZRBQU8n8Kcn6+2X9/ape/1BaZk9M7445SZIkSRqPXi5gKkmSJEmTZOMjSZIkaeZNdeMTERsj4t6I2BkRl/ddz1JExFURsTsi7l4wdnJE3BQRn+t+ntRnjQcTEadHxC0RsT0i7omIN3XjVepfHRGfjIhPdfX/1258fUTc2n2P/qg7UW5qRcTKiLgjIv6s2y5Tf0R8MSI+HRF37lufv8r3ByAiToyI6yLiMxGxIyLOr1S/xsf8mSzzZzqYP/0xfyZjahufiFgJXAm8EjgLuDQizuq3qiV5L7D/2uaXAzdn5pnAzd32NNoL/FxmngW8GPjp7r95lfqfBF6emWcD5wAbI+LFwDuAd2bmc4HHgMt6rHEp3gTsWLBdrf6XZeY5C649UOX7A/Au4C8z8/nA2Qz+P1SqX2Ng/vTC/JkO5k9/zJ9JyMypvAHnAzcu2H4r8Na+61pi7WcAdy/Yvhc4tbt/KnBv3zUu8XNcD7yiYv3AscDfMbjS7yPAqm78ad+rabsxWKv+ZuDlwJ8BUaz+LwJr9hsr8f1hcA2AL9At+lKtfm9j/S6YP/1/DvNn8nWbP/3Vbv5M6Da1R3yA04AHFmzv6sYqWpuZD3X3HwbW9lnMUkTEGcC5wK0Uqr87TH8nsBu4Cfg88Hhm7u2eMu3fo98CfgGY77ZPoVb9CXwoIm6PiM3dWJXvz3rgK8Dvd1M93h0Rx1Gnfo2P+dMj86c35k9/zJ8JmebGZybloG2f6jXEI+J44I+BN2fmPyx8bNrrz8y5zDyHwZ6r84Dn91zSkkXEq4HdmXl737WM4KWZ+SIGU4R+OiL+5cIHp/z7swp4EfC7mXku8HX2m1Yw5fVLh1Th+2v+9MP86Z35MyHT3Pg8CJy+YHtdN1bRlyPiVIDu5+6e6zmoiDiKQei8LzP/pBsuU/8+mfk4cAuDQ/MnRsS+i/VO8/foJcBFEfFF4BoG0w3eRZ36ycwHu5+7gQ8yCP8q359dwK7MvLXbvo5BEFWpX+Nj/vTA/OmV+dMv82dCprnxuQ04s1tR5GgGV2Td2nNNw9oKbOrub2Iwd3nqREQwuBLujsz8zQUPVan/2yLixO7+MxjMD9/BIIBe2z1tauvPzLdm5rrMPIPB9/3DmfmjFKk/Io6LiBP23Qd+GLibIt+fzHwYeCAivrsbuhDYTpH6NVbmz4SZP/0yf/pl/kxODI6cTaeIeBWDOacrgasy81d7LumwIuIDwAXAGuDLwNuBPwWuBZ4D3A9ckplf7avGg4mIlwJ/DXyaf5rj+4sM5llXqP+FwNUMvi8rgGsz84qI+C4Ge7BOBu4Afiwzn+yv0sOLiAuAn8/MV1epv6vzg93mKuD9mfmrEXEKBb4/ABFxDvBu4GjgPuAn6L5LFKhf42P+TJb5Mz3Mn36YP5Mx1Y2PJEmSJI3DNE91kyRJkqSxsPGRJEmSNPNsfCRJkiTNPBsfSZIkSTPPxkeSJEnSzLPxkSRJkjTzbHwkSZIkzTwbHy1rEfF9EXFXRKzurvx8T0R8b991SZJmm/kjTZ4XMNWyFxG/AqwGngHsysz/3nNJkqRlwPyRJsvGR8teRBwN3AZ8E/iBzJzruSRJ0jJg/kiT5VQ3CU4BjgdOYLDnTZKkSTB/pAnyiI+WvYjYClwDrAdOzcw39lySJGkZMH+kyVrVdwFSnyLix4GnMvP9EbES+FhEvDwzP9x3bZKk2WX+SJPnER9JkiRJM89zfCRJkiTNPBsfSZIkSTPPxkeSJEnSzLPxkSRJkjTzbHwkSZIkzTwbH0mSJEkzz8ZHkiRJ0sz7/6xgnhFO0jFqAAAAAElFTkSuQmCC\n" + }, + "metadata": { + "needs_background": "light" + } + } + ], + "source": [ + "vis.plot(generate_heat_example(batch(view_examples=2), spatial(x=64, y=64)));" + ], + "metadata": { + "pycharm": { + "name": "#%%\n" + }, + "colab": { + "base_uri": "https://localhost:8080/", + "height": 390 + }, + "id": "BHtnPdL3S2eO", + "outputId": "819f73cc-1346-4243-94cc-8416e7c8c621" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "## Differentiable physics and gradient descent\n", + "\n", + "Nothing in this setup so far prevents us from using regular DP training, as described in {doc}`diffphys`.\n", + "For this diffusion case, we can write out\n", + "gradient descent update from in an analytic fashion as:\n", + "$$\\Delta x_{\\text{GD}} = - \\eta \\cdot \\mathcal F^{-1}\\left( e^{-k^2 (t_* - t_0)} \\mathcal F(y - y^*) \\right).$$\n", + "\n", + "Looking at this expression, it means that gradient descent (GD) with the gradient from the differentiable simulator applies the forward physics to the gradient vector itself. This is surprising: the forward simulation performs diffusion, and now the backward pass performs even more of it, instead of somehow undoing the diffusion? Unfortunatley, this is the inherent and \"correct\" behavior of DP, and it results in updates that are stable but lack high frequency spatial information.\n", + "\n", + "Consequently, GD-based optimization methods converge slowly on this task after fitting the coarse structure and have severe problems in recovering high-frequency details, as will be demonstrated below.\n", + "This is not because the information is fundamentally missing but because GD cannot adequately process high-frequency details.\n", + "\n", + "For the implementation below, we will simply use `y = diffuse.fourier(x, 8., 1)` for the forward pass, and then similarly compute an $L^2$ loss for two $y$ fields to which `diffuse.fourier( , 8., 1)` is applied.\n", + "\n" + ], + "metadata": { + "collapsed": false, + "id": "XURdC4v_S2eP" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Stable SIP gradients\n", + "\n", + "What is more interesting in the context of this chapter is the improved update step computed via the inverse simulator, the _SIP_ update. In line with the previous sections, we'll call this update $\\Delta x_{\\text{PG}}.$\n", + "\n", + "The frequency formulation of the heat equation can be inverted analytically, yielding\n", + "$\\hat x_k = \\hat y_k \\cdot e^{k^2 (t_* - t_0)}$.\n", + "This allows us to define the update \n", + "$$\\Delta x_{\\text{PG}} = - \\eta \\cdot \\mathcal F^{-1}\\left( e^{k^2 (t_* - t_0)} \\mathcal F(y - y^*) \\right).$$\n", + "Here, high frequencies are multiplied by exponentially large factors, resulting in numerical instabilities.\n", + "When applying this formula directly to the gradients, it can lead to large oscillations in $\\Delta x_{\\text{PG}}$.\n", + "\n", + "Note that these numerical instabilities also occur when computing the gradients in real space instead of frequency space.\n", + "However, frequency space allows us to more easily quantify them.\n", + "\n", + "Now we can leverage our knowledge of the physical simulation process to construct a stable inverse:\n", + "the numerical instabilities can be avoided by taking a probabilistic viewpoint.\n", + "The observed values $y$ contain a certain amount of noise $n$, with the remainder constituting the signal $s = y - n$.\n", + "For the noise, we assume a normal distribution $n \\sim \\mathcal N(0, \\epsilon \\cdot y)$ with $\\epsilon > 0$ and for the signal, we assume that it arises from reasonable values of $x$ so that $y \\sim \\mathcal N(0, \\delta \\cdot e^{-k^2})$ with $\\delta > 0$.\n", + "With this, we can estimate the probability of an observed value arising from the signal using Bayes' theorem\n", + "$p(s | v) = \\frac{p(v | s) \\cdot p(s)}{p(v | s) \\cdot p(s) + p(v | n) \\cdot p(n)}$ where we assume the priors $p(s) = p(n) = \\frac 1 2$.\n", + "Based on this probability, we dampen the amplification of the inverse physics which yields a stable inverse.\n", + "\n", + "Gradients computed in this way hold as much high-frequency information as can be extracted given the noise that is present.\n", + "This leads to a much faster convergence and more precise solution than any generic optimization method.\n", + "The cell below implements this probabilistic approach, with `probability_signal` in `apply_damping()` containing the parts of the signal to be dampened.\n", + "The `inv_diffuse()` functions employs it to compute a stabilized inverse diffusion process.\n" + ], + "metadata": { + "id": "EUIM1Av5vlfR" + } + }, + { + "cell_type": "code", + "execution_count": 146, + "outputs": [], + "source": [ + "def apply_damping(kernel, inv_kernel, amp, f_uncertainty, log_kernel):\n", + " signal_prior = 0.5\n", + " expected_amp = 1. * kernel.shape.get_size('x') * inv_kernel # This can be measured\n", + " signal_likelihood = math.exp(-0.5 * (abs(amp) / expected_amp) ** 2) * signal_prior # this can be NaN\n", + " signal_likelihood = math.where(math.isfinite(signal_likelihood), signal_likelihood, math.zeros_like(signal_likelihood))\n", + " noise_likelihood = math.exp(-0.5 * (abs(amp) / f_uncertainty) ** 2) * (1 - signal_prior)\n", + " probability_signal = math.divide_no_nan(signal_likelihood, (signal_likelihood + noise_likelihood))\n", + " action = math.where((0.5 >= probability_signal) | (probability_signal >= 0.68), 2 * (probability_signal - 0.5), 0.) # 1 sigma required to take action\n", + " prob_kernel = math.exp(log_kernel * action)\n", + " return prob_kernel, probability_signal\n", + "\n", + "\n", + "def inv_diffuse(grid: Grid, amount: float, uncertainty: Grid):\n", + " f_uncertainty: math.Tensor = math.sqrt(math.sum(uncertainty.values ** 2, dim='x,y')) # all frequencies have the same uncertainty, 1/N in iFFT\n", + " k_squared: math.Tensor = math.sum(math.fftfreq(grid.shape, grid.dx) ** 2, 'vector')\n", + " fft_laplace: math.Tensor = -(2 * np.pi) ** 2 * k_squared\n", + " # --- Compute sharpening kernel with damping ---\n", + " log_kernel = fft_laplace * -amount\n", + " log_kernel_clamped = math.minimum(log_kernel, math.to_float(math.floor(math.log(math.wrap(np.finfo(np.float32).max))))) # avoid overflow\n", + " raw_kernel = math.exp(log_kernel_clamped) # inverse diffusion FFT kernel, all values >= 1\n", + " inv_kernel = math.exp(-log_kernel)\n", + " amp = math.fft(grid.values)\n", + " kernel, sig_prob = apply_damping(raw_kernel, inv_kernel, amp, f_uncertainty, log_kernel)\n", + " # --- Apply and compute uncertainty ---\n", + " data = math.real(math.ifft(amp * math.to_complex(kernel)))\n", + " uncertainty = math.sqrt(math.sum(((f_uncertainty * kernel) ** 2))) / grid.shape.get_size('x') # 1/N normalization in iFFT\n", + " uncertainty = grid * 0 + uncertainty\n", + " return grid.with_values(data), uncertainty, abs(amp), raw_kernel, kernel, sig_prob" + ], + "metadata": { + "pycharm": { + "name": "#%%\n" + }, + "id": "jJFlJlRwS2eP" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Neural network and loss function\n", + "\n", + "For the neural network, we use a simple U-net architecture for the SIP and the regular DP+Adam version (in line with the previous sections, we'll denote it as `GD`). \n", + "We train with a batch size of 128 and a constant learning rate of $\\eta = 10^{-3}$, using 64 bit precision for physics but 32 bits for the network.\n", + "The network updates are computed with TensorFlow's or PyTorch's automatic differentiation.\n", + "\n" + ], + "metadata": { + "collapsed": false, + "id": "Prft7mS6S2eQ" + } + }, + { + "cell_type": "code", + "execution_count": 147, + "outputs": [], + "source": [ + "math.set_global_precision(64)\n", + "BATCH = batch(batch=128)\n", + "STEPS = 50\n", + "\n", + "math.seed(0)\n", + "net = u_net(1, 1)\n", + "optimizer = adam(net, 0.001)" + ], + "metadata": { + "pycharm": { + "name": "#%%\n" + }, + "id": "edLI4HnjS2eR" + } + }, + { + "cell_type": "markdown", + "source": [ + "Now we'll define loss function for phiflow.\n", + "The gradients for the network weights are always computed as d(loss_function)/d$\\theta$.\n", + "For SIP, we invert the physics, and then define the proxy loss as $L^2$(prediction - correction), where correction is taken as constant. Below this is realized via `x = field.stop_gradient(prediction)`.\n", + "\n", + "The Adam / GD version for `sip=False` simply computes the L2 difference of two diffused $y$ fields for the gradient, as outlined above.\n", + "\n", + "We also compute the reference $y = \\mathcal P(x^*)$ in the loss function on-the-fly.\n" + ], + "metadata": { + "collapsed": false, + "id": "lkkXZc_kS2eR" + } + }, + { + "cell_type": "code", + "execution_count": 148, + "outputs": [], + "source": [ + "# @math.jit_compile\n", + "def loss_function(net, x_gt: CenteredGrid, sip: bool):\n", + " y_target = diffuse.fourier(x_gt, 8., 1)\n", + " with math.precision(32):\n", + " prediction = field.native_call(net, field.to_float(y_target)).vector[0]\n", + " prediction += field.mean(x_gt) - field.mean(prediction)\n", + " x = field.stop_gradient(prediction)\n", + " if sip:\n", + " y = diffuse.fourier(x, 8., 1)\n", + " dx, _, amp, raw_kernel, kernel, sig_prob = inv_diffuse(y_target - y, 8., uncertainty=abs(y_target - y) * 1e-6)\n", + " correction = x + dx\n", + " loss = field.l2_loss(prediction - correction)\n", + " y_l2 = field.l2_loss(y - y_target)\n", + " else:\n", + " y = diffuse.fourier(prediction, 8., 1)\n", + " y_l2 = loss = field.l2_loss(y - y_target)\n", + " return loss, x, y, field.l2_loss(x_gt - x), y_l2\n" + ], + "metadata": { + "pycharm": { + "name": "#%%\n" + }, + "id": "aUKHZMwkS2eR" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Training\n", + "\n", + "In the training loop, we generate data on-the-fly via `generate_heat_example()` and use phiflow's `update_weights()` function to call the correct functions of the chosen backend. Note that we're only printing the loss of the first 5 steps below for clarity, the whole history is collected in the `loss_` lists. The following cell runs the SIP version of the training:\n" + ], + "metadata": { + "collapsed": false, + "id": "0L142o1OS2eS" + } + }, + { + "cell_type": "code", + "source": [ + "loss_sip_x=[]; loss_sip_y=[] \n", + "for training_step in range(STEPS):\n", + " data = generate_heat_example(spatial(x=64, y=64), BATCH)\n", + " loss_value, x_sip, y_sip, x_l2, y_l2 = update_weights(net, optimizer, loss_function, net, data, sip=True)\n", + " loss_sip_x.append(float(x_l2.mean)) \n", + " loss_sip_y.append(float(y_l2.mean))\n", + " if(training_step<5): print(\"SIP L2 loss x ; y: \"+format(float(x_l2.mean))+\" ; \"+format(float(y_l2.mean)) )" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "PK47PUNKaeA0", + "outputId": "2529bd34-0594-443e-8f71-9e1fe72d1435" + }, + "execution_count": 149, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:3635: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.\n", + " \"See the documentation of nn.Upsample for details.\".format(mode)\n" + ] + }, + { + "output_type": "stream", + "name": "stdout", + "text": [ + "SIP L2 loss x ; y: 187.2586057892786 ; 59.48433060883144\n", + "SIP L2 loss x ; y: 70.21347147390776 ; 13.783476203544797\n", + "SIP L2 loss x ; y: 51.91472605336263 ; 5.72813432496525\n", + "SIP L2 loss x ; y: 39.46109317565444 ; 4.4424629873554045\n", + "SIP L2 loss x ; y: 34.611490797378366 ; 3.359133206049436\n" + ] + } ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" + }, + { + "cell_type": "markdown", + "source": [ + "And now we can repeat the training with the DP version using Adam, deactivating the SIP update via `sip=False`:" + ], + "metadata": { + "id": "32HFjc2y7noA" + } + }, + { + "cell_type": "code", + "execution_count": 150, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:3635: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.\n", + " \"See the documentation of nn.Upsample for details.\".format(mode)\n" + ] + }, + { + "output_type": "stream", + "name": "stdout", + "text": [ + "GD L2 loss x ; y: 187.2586057892786 ; 59.48433060883144\n", + "GD L2 loss x ; y: 104.37856249315794 ; 20.323700695817763\n", + "GD L2 loss x ; y: 72.5135221242247 ; 7.211550418534284\n", + "GD L2 loss x ; y: 59.74792697261851 ; 5.3912096056107135\n", + "GD L2 loss x ; y: 50.49939087445511 ; 3.6758429757536093\n" + ] + } + ], + "source": [ + "math.seed(0)\n", + "net_gd = u_net(1, 1)\n", + "optimizer_gd = adam(net_gd, 0.001)\n", + "\n", + "loss_gd_x=[]; loss_gd_y=[] \n", + "for training_step in range(STEPS):\n", + " data = generate_heat_example(spatial(x=64, y=64), BATCH)\n", + " loss_value, x_gd, y_gd, x_l2, y_l2 = update_weights(net_gd, optimizer_gd, loss_function, net_gd, data, sip=False)\n", + " loss_gd_x.append(float(x_l2.mean)) \n", + " loss_gd_y.append(float(y_l2.mean))\n", + " if(training_step<5): print(\"GD L2 loss x ; y: \"+format(float(x_l2.mean))+\" ; \"+format(float(y_l2.mean)) )" + ], + "metadata": { + "pycharm": { + "name": "#%%\n" + }, + "id": "0ENSdgZiS2eS", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "0b3f0a60-aa3a-4b5a-9e6d-dd4fef70f509" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Evaluation\n", + "\n", + "Now we can evaluate how the two variants behave in direct comparison. Note that due to the on-the-fly generation of randomized data, all samples are previously unseen, and hence we'll directly use the training curves here to draw conclusions about the performance of the two approaches.\n", + "\n", + "The following graph shows the $L^2$ error over training in terms of the reconstructed $x$ input, the main goal of our training." + ], + "metadata": { + "id": "eV1j8lYuwUqB" + } + }, + { + "cell_type": "code", + "source": [ + "import pylab as plt\n", + "fig = plt.figure().gca()\n", + "pltx = range(len(loss_adam_x)) # np.linspace(-1,1,N)\n", + "fig.plot(pltx, loss_adam_x , lw=2, color='blue', label=\"GD\") \n", + "fig.plot(pltx, loss_sip_x , lw=2, color='red', label=\"SIP\")\n", + "plt.xlabel('iterations'); plt.ylabel('x loss'); plt.legend(); plt.yscale(\"log\")\n" + ], + "metadata": { + "id": "uUkHS8Xljn7O", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 300 + }, + "outputId": "3dd0ee6e-3592-4d03-d5ed-a886d7994b6e" + }, + "execution_count": 151, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEGCAYAAABlxeIAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hUZdoG8PtJAoQSQgkg0gIiCggSmiLFgMoiLitYwLKKu7iuwqpgYdX1w7Ki6OoKlrVhQVdQV0UR6U1AkA5RihTpnQCBEAIheb4/npnMpEwySaZm7t91nWtmzpzyHpic57xdVBVERERRwU4AERGFBgYEIiICwIBAREQODAhERASAAYGIiBxigp2AskhISNDExMRgJ4OIKKysWrXqiKrWyb8+rANCYmIiVq5cGexkEBGFFRHZWdh6FhkREREABgQiInJgQCAiIgBhWocgIv0A9GvevHmwk0JE5UxWVhb27NmDzMzMYCelzGJjY9GwYUNUqFDBq+0lnMcy6tixo7JSmYh8afv27YiLi0Pt2rUhIsFOTqmpKlJTU3Hy5Ek0bdo0z3ciskpVO+bfh0VGRERuMjMzwz4YAICIoHbt2iXK6TAgEBHlE+7BwKmk1xFxAeHgz4ewuk5vrKjbN9hJISIKKREXEKomVEb7I7PR6vAPyM4OdmqIiAp38OBB3HbbbWjWrBk6dOiALl26YPLkyViwYAHi4+ORlJSEiy66CD169MDUqVN9cs6wbGVUFtXOq4bTUhlVNQM7NqYj8ZJqwU4SEVEeqor+/ftj8ODBmDhxIgBg586dmDJlCmrWrInu3bvnBoG1a9eif//+qFy5Mq666qoynTficggQQVrFugCA3asOBTkxREQFzZs3DxUrVsS9996bu65Jkya4//77C2zbrl07jBo1Cm+88UaZzxt5AQHA6eoWEA6kMCAQkWci/lmKs379erRv397rdLZv3x6bNm0qw5WaiAwIObUtIBzfzIBARKFv2LBhuPTSS9GpU6dCv/dVf7KIDAgxDeoBADJ2HAxySogolKn6ZylO69atsXr16tzPb775JubOnYvDhw8Xuv2aNWvQsmXLMl9vRAaEKomWQzi3jzkEIgo9vXr1QmZmJt56663cdRkZGYVum5KSgn/+858YNmxYmc8bca2MAKBGCwsIFY4dQlYW4OUwH0REASEi+OabbzBixAi89NJLqFOnDqpWrYoXX3wRALBo0SIkJSUhIyMDdevWxWuvvVbmFkZAhAaECg0sINTRQ9i+HWjRIsgJIiLKp379+vjss88K/S4tLc0v54zIIiPUszqEejiIzZuDnBYiohARmQGhruUQ6uIQtmwJclqIiEIEAwIDAhERgEgNCAkJUBEk4Ai2beaARkREQKQGhJgYZNeojSgojmw6EuzUEBGFhMgMCACiz7Nio6y9h1AOZsojIiqziA0IUs/R9BSHsG1bkBNDRJTP6NGj0bp1a7Rt2xbt2rXDsmXLkJycDOe0wYmJiWjTpg3atm2L3r1748CBA2U+Z8QGBPeKZTY9JaJQsnTpUkydOhWrV69GSkoK5syZg0aNGhXYbv78+UhJSUHHjh3x/PPPl/m8kRsQ3PoisKUREYWS/fv3IyEhAZUqVQIAJCQk4Pzzz/e4fY8ePbB169YynzdyAwKbnhJRcYI0/nXv3r2xe/dutGjRAkOHDsUPP/xQ5PZTp05FmzZtyny5DAgsMiKiEFOtWjWsWrUK7777LurUqYNBgwbho48+KrBdz5490a5dO5w4cQKPP/54mc8bkWMZAWAOgYiK56N5BkojOjoaycnJSE5ORps2bTBhwoQC28yfPx8JCQk+O2fk5hAcdQjnyUHs3w+kpwc5PUREDr/++iu2uD2prl27Fk2aNPH7eSM3IDhyCOfH2JwIzCUQUahIT0/H4MGD0apVK7Rt2xYbNmzA008/7ffzRnyRUUKOKyAkJQUzQUREpkOHDliyZEmB9QsWLMh9v2PHDp+fN3JzCNWqAbGxiM3OQBWcYg6BiCJe5AYEEfZFICJyE7kBAWDTUyIqlAaxdZEvlfQ6GBDApqdE5BIbG4vU1NSwDwqqitTUVMTGxnq9T+RWKgO5AaFhxUM4cgQ4dgyoWTPIaSKioGrYsCH27NmDw4cPBzspZRYbG4uGDRt6vT0DAoBWtQ4CB6ylUefOQU4TEQVVhQoV0LRp02AnIygiu8jIUancrBr7IhARRXZAcOQQGlRkQCAiYkAAUEctILClERFFMgYEAPGZBwEwh0BEkS2yA4KjDqHySVeRUZi3NCMiKrXIDgiOYWMl9QhqVs9GWhpQDlqaERGVSmQHhJgYoHZtiCo6Nk0FwGIjIopckR0QgNx6hPYNWI9ARJGNAcFRj9CyNpueElFkY0Bw5BAuiGPTUyKKbAwIjoDQOJY5BCKKbAwIzhFP1VWHwKanRBSJGBAcdQixJw4hIQHIyAD27AlymoiIgiBkAoKI9BeR90TkcxHpHbATO3IIOHQI7drZ2+XLA3Z2IqKQ4deAICIfiMghEfkl3/o+IvKriGwVkccAQFW/UdW/ALgXwCB/pisPt4DQrZu9Xbw4YGcnIgoZ/s4hfASgj/sKEYkG8CaAawG0AnCriLRy2+RJx/eB4QwIBw/mBoQffwzY2YmIQoZfA4KqLgRwNN/qzgC2qupvqnoWwGcArhfzIoDpqrra0zFF5B4RWSkiK30yo5GjDgGHDuGyy4DoaGD1auDUqbIfmogonASjDqEBgN1un/c41t0P4GoAN4nIvZ52VtV3VbWjqnasU6dO2VNTrRoQGwtkZKCanEJSEpCdDSxbVvZDExGFk5CpVFbV11S1g6req6pvB+zEIqxHICJCcALCXgCN3D43dKwLHrd6hK5d7S0DAhFFmmAEhBUALhSRpiJSEcAtAKYEIR0ubvUIzoCwdClw7lzwkkREFGj+bnY6CcBSABeJyB4RGaKq5wD8DcBMABsBfKGq6/2ZjmK5FRnVrw9ccAGQng6kpAQ1VUREARXjz4Or6q0e1k8DMM2f5y4Rt4AAAN26Adu2WbFR+/ZBTBcRUQCFTKVyULnVIQBgxTIRRaSwDAgi0k9E3k1LS/PNAd3qEIC8AYED3RFRpAjLgKCq36nqPfHx8b45YL4io4suAmrXBvbvB7Zv980piIhCXVgGBJ/LFxBEkNvaiMNYEFGkYEAACtQhAKxHIKLIw4AAAM4hMI4csXErwIBARJGHAQEAYmKs0kAVSE0FYM1NY2OBDRtyVxERlWsMCE756hEqVQI6d7ZVS5YEKU1ERAHEgODEegQiinBhGRB83g8BKNAXAWBAIKLIEpYBwef9EIACRUYA0KWLNUFdsQI4fdp3pyIiCkVhGRD8opCAUKMGcMklQFYWsHJlkNJFRBQgDAhOhdQhACw2IqLIwYDgVEgdAsCAQESRgwHBqZAiI8AVEJYsAXJyApwmIqIAYkBw8hAQGjcGGjUCjh+3TmpEROUVA4KThzoEAOjRw16/+iqA6SEiCjAGBKe4OBurIiMDOHUqz1dDhtjrW28BZ84EIW1ERAHAgOAk4sol7N2b56vkZKBtW8s8TJoU+KQREQVCWAYEv/RUBlwTKC9YkO98wIgR9n7sWM6iRkTlU1gGBL/0VAaAPn3sdcaMAl/dcotlINatKxAviIjKhbAMCH7jDAhz5lj3ZDexscDQofZ+7NgAp4uIKAAYENw1aQK0bAmcPAksXVrg6/vuAypWBL77Dti6NQjpIyLyIwaE/Jy5hOnTC3xVty5w++1WhzBuXIDTRUTkZwwI+RVRjwC4Kpc//NA6qxERlRcMCPn16AFUrgysXQvs31/g6zZtgKuusq4K48cHIX1ERH7CgJBfbKx1PACAWbMK3WT4cHt9/XXg3LnAJIuIyN8YEApz7bX26qHYqG9foEULYNcuYPLkAKaLiMiPGBAK46xHmDULyM4u8HVUFPDgg/b+1VcDmC4iIj9iQChM8+ZAs2bA0aM2f2Yh7rzTZlRbuhRYtizA6SMi8oOwDAh+G7rCdYJiWxtVqwbcc4+9HzPGP8kgIgqksAwIfhu6wl0x9QiAVS7HxgLffAOsWeO/pBARBUJYBoSASE62bsnLlwNHjhS6Sf36ruEsnn46YCkjIvILBgRPqlUDune3bsmzZ3vcbORIoEoVYMoUYOXKAKaPiMjHig0IIvKgiFQX876IrBaR3oFIXNAVU48AAPXqAX/7m71/6qkApImIyE+8ySH8WVVPAOgNoCaAOwBERjWqsx5h5kwgJ8fjZo8+ClStCkybBvz0U4DSRkTkY94EBHG89gXwiaqud1tXvrVqBTRsaFOlrVvncbOEBOCBB+w96xKIKFx5ExBWicgsWECYKSJxADw/Lpcn7s1PCxn91N3DD9u0zDNnAj/+GIC0ERH5mDcBYQiAxwB0UtUMABUA/MmvqQolXtQjAEDt2q4xjliXQEThyJuA0AXAr6p6XET+COBJAH7qERaCrr4aiI4GliwBiukIN2IEEB8PzJ0LLFwYoPQREfmINwHhLQAZInIpgIcBbAPwsV9TFUri44ErrrAxjYpofgoANWu65ktgLoGIwo03AeGcqiqA6wG8oapvAojzb7JCzHXX2euUKcVuOny4jXG0YAEwf75/k0VE5EveBISTIvI4rLnp9yISBatHiBz9+9vr1KlAVlaRm8bHA488Yu+HDfPYyZmIKOR4ExAGATgD649wAEBDAP/ya6pCzUUX2XLsGLB4cbGbP/CAtVjduBHo3ZtTbRJReCg2IDiCwKcA4kXk9wAyVTWodQh+H+20MM5cwjffFLtpXBwwZ46Nor1mjfVvO3nSz+kjIiojb4auGAhgOYCbAQwEsExEbvJ3wooSkNFO87v+env99lsb36gY9etba6PGja33cr9+QEaGn9NIRFQG3hQZ/QPWB2Gwqt4JoDOA//NvskLQZZfZwEU7dxbZa9ld48bAvHnA+ecDP/wADBgAnDnj53QSEZWSNwEhSlUPuX1O9XK/8iUqCvjDH+z9t996vdsFF1hOoU4dm5Fz4MBi66WJiILCmxv7DBGZKSJ3ichdAL4HMM2/yQpRJahHcHfxxVanULOmtVzt1Qt46SXrvHbqlB/SSURUCqJelIeLyI0Aujo+LlLVyX5NlZc6duyoKwM5CUFmpo1kd+oUsGMH0KRJiXZfuRK46irgxAnXuuho4JJLgMsvtxZJN9zg2yQTEeUnIqtUtWOB9d4EhFAV8IAAADffDHz5JTBunGuI0xLYv9+KjpYts8rmlBTrBO00f75N1kZE5C+eAoLHIiMROSkiJwpZTorICU/7lXvO1kYlLDZyql8fGDwY+M9/gNWrLbewcCFw6632/Rtv+CidREQlxBxCSR07ZjXEAHDoEFCrlk8Ou3+/tUpSBbZvBxo18slhiYgKKHEOgTyoWdPKdLKzge+/99lh69cHbrzRDvvOOz47LBGR1xgQSsO9k5oPOedmfvdd9lcgosBjQCgNZ0CYMcNaHvlI167ApZcChw8D//ufzw5LROQVb4auaFXIumS/pCZcNG4MJCVZ89O5c312WBFXLoGVy0QUaN7kEL4Qkb+LqSwirwN4wd8JC3ml7KRWnNtus/kUli2zfgtERIHiTUC4DEAjAEsArACwD65OapHLWWw0ZUrejgRlVKUKMGSIvX/zTZ8dloioWN4EhCwApwFUBhALYLuq5vg1VeGgbVsgMdGani5b5tND33efFR9NmsQJdogocLwJCCtgAaETgO4AbhWRoFZ5BmU+hIKJcOUS3n/fp4e+4AKgb19raeTjQxMReVRsxzQR6aiqK/Otu0NVP/FryrwQlI5p7jZutGZBWVk+H3NixgybWKdxY+C332zMIyIiXyh1x7T8wcCxLujBICS0bAk8+aS9v/tun86A07u3zbi2a5dN5UxE5G/sh1BWjz1mw5Vu2wY89ZTPDhsVBQwdau/ZBJWIAoEBoawqVgQ++MDu4P/+N7Bihc8Ofddd1upozhxgwwafHZaIqFAMCL7QqRPw0ENATg7w5z8DZ8/65LA1awJ33mnvBw/2aadoIqICGBB85ZlnrND/l1+AMWN8dtjRo61168qVwP33++ywREQFMCD4SpUqwHvv2fvnnrPA4AO1agFffw3ExgLjx9tCROQPDAi+lJwM/PWv1gx1yBCf9WBOSgLeftveDxvm02oKIqJcDAi+9uKLQIMGwPLl1gLJRwX/gwdbD+azZ4GbbmIPZiLyPQYEX4uPd81w8/LLVq/w9ts+qWgeOxa4/HLrm3DrrT4dQomIiAHBL667Dpg+3Xox791rj/YXXwxMmFCmu3jFijZPQt261hTV2SeOiMgXGBD8pU8fYPVqu4O3bGkTJd91l3Vi++STUvdqbtgQ+PxzG8pizBjgyy99m2wiilwMCP4UFWUF/j//DHz8MdCsGbBpk3UuqF8fuOceYMkSoJjxpPJLTraqCgC4/XZg3jzfJ52IIg8DQiBERwN33GHBYPx4oHNn4MQJa6batStw0UXA889b8ZKXHnrIZlc7e9YGXWXLIyIqq2JHOw1lQR/ttCw2bLA6hU8+Afbvt3XVqwOrVllFtBdycizOTJwI1K4NLFwItCow4WnxMjOBgwdtZFWRku9PROGl1KOdkp+0amXlPrt2AdOmAVdcYbmGV17x+hBRUcBHH1kddmqqjZC6c6f3SUhPt4ZQiYm21K0L3Hgj8NprwLp1FnCIKHIwhxAqNm60IBEba0GiTh2vdz192uqwFy4ELrwQWLQIqFfP8/ZpaTaC6quvWiABgLg44OTJvNvVrAn07Am89JJN2kNE5QNzCKGuZUt71M/MBN56q0S7Vq5sUzsnJQFbtlhw2LnTSqJ27bIJdn79FUhJAUaNApo0sSarqalAly6WQUlLs33Hj7diqMaNgWPHbNiM3r1tplAiKt+YQwglCxbYI3mdOnZHr1y5RLsfOgR07w5s3lz8tj17WlDo2dNzvcH27cDNN1u1xmWXWWumKlVKlCQiCkHMIYSDK68E2rcHDh8G/vvfEu9ety4we7Y99deuDZx3nvVbaNrUipJatbIWSYsX2829V6+iK5GbNrXZ2po0AZYtA/74R/aOJirPwjKHICL9APRr3rz5X7Zs2RLs5PjWpEnAbbdZU9QNG6zmOMg2bLA677Q0YPhwq3sgovBVrnIIqvqdqt4THx8f7KT43k03AY0aWaH/tGnBTg0Ay1lMngxUqGDjKY0bF+wUEZE/hGVAKNcqVLDHcMDahIaInj2BDz+09yNGWIAgovIlLIuMnMpdpbLTiROWSzhxwrogdyyQswua0aOtMjo2FnjiCatfOP981xIfb/USWVnA8ePWUsn5WrEi0KYNkJAQ7KsgimyeioxigpEYKkb16jbO0csvW0e1SZOCnaJcTzwB7NhhzVNHjSr4fZUqFhBOnfJ8jPPOA9q2teDQtq21YLroIr8lmYi8xBxCqNq92wbDUwW2bbNH8RBx7hzw6ac2S+i+fa5l715XIIiKAmrUsM5tztf0dNsnPb3gMUePBh5/nENnEAUCcwjhplEjYNAgu/OOGwf8+9/BTlGumBibwS0/VVdv57i4wm/uOTmWw/j5Z+sot3at1Uf84x/W2vaVV0KiYRVRRGIOIZStWWP9EqpVsxxDjRrBTpFffPGF9XHIyrLXDz6wuvXipKdbp7nly21Ztcr6XTz3HNCjh//TTRSuPOUQGBBC3VVXWS+y3/0O+OoroGrVYKfIL2bPBgYMsCKn666zIJG/V/S5c8D8+fbPsGQJsH695wH4+ve3sQNbtPB/2onCTbnqhxBRxo2zoSxmzgSuuQY4ejTYKfKLa66xuFe7NvD99zZ+0rFjdsNfvNjmfmjQwNa/844VOUVFWQbq3nstV7F2LfD00xZIvvkGaN0auP9+4MiRYF8dUXhgDiEcbN5sd8xdu2wKzpkzrY1nObRxo9309+yxaSEyM+29U4sWwC23WIYpKanw4Z7277cWUB98YAGlenVg2DALEE2a2MB9559vdSHh4tw5m3SvVSvg8suDnRoKdywyCne7d9udctMmG2Ro9uxyOya1+6UCdgO/5RZb2rXzviXSzz8DI0cCM2YU/C4qynIcLVpYR7u+fUO3hdPBg3btCxZYcNu2jX05qGwYEMqDI0eAa68FVq60xvwzZ1pD/nIoNdUmk+vc2Z6Iy9LyaO5cYPp0CzS7dtmyf3/eqayvuspaOF16adnT7m7dOqvw7t7d+lqUNOgsXWqjmezb51r38MPedWLfutWK4QYNsg6DRE6eAgJUNWyXDh06aMQ5cUK1Vy9VQDU+XnXx4rId78gR1T/+UbVNG9WdO32TxjCQmam6bZvqK6+o1qhh/5wiqn/6k+revWU/fna26gsvqEZH27EB1ebNVYcPV507V/Xs2aL3z8lRff111QoVbN9u3VS/+87eV6qkunt30fufPKmamGjb166t+uqrds2+lpWlevq0749L/gVgpRZyTw36Tb0sS0QGBFX7CxwwwP77qlRRnT27dMeZMUO1fn3XHeu++3ybzjCRmmo3aufNt0oV1VGjVDMySne8AwdUe/d2/bP27m03ZednQLV6ddUbb1R99lnVzz5TXb1aNT3d9k9PV739dte2w4e7AsjAgbbu7ruLTsOIEbZdbKzrOImJqp9+asHKFw4eVG3Xzs7x17+qbtzom+OS/zEglDdZWaqDB9t/YcWKqlOmeL/vqVOqw4a57hQdO7oePQ8c8FuSQ92WLao33OD6Z7n6atUzZ0p2jDlzVM87z/ZPSFD9/ntbf+6cZeYee0y1deu8wcF9adBAtWFDe1+1qgULd7/+armOqCjPN+Bly+z7qCjVFSvsp9GqlescSUmqs2aV/N/H3b59qi1bFkx/3772b5CTU7bjFyc7W3XMGNU//9kVSMl7DAjlUXa26tCh9t8YE1Pw7lGYZctUW7SwfSpUsL+qc+dU+/e3dY8/7v90h7iFC1Xr1rV/jjvv9O7mlpWl+o9/WLEToJqcrLpnj+ftf/tN9f33VUeOtH/6Vq0srjtvrC1aqP7yS+H73nOPbXPTTQW/O3NG9ZJL7PtHHsmbvvfft4DjPMfAgaqHDxd/bfnt3q164YV2jDZtVBctshyCe26kbVvVd9+1gHT0aMnPUZS0NNV+/VzneuIJ3x4/EjAglFc5Oap//7vmFoKPH19wm+xs1R9/VH3gAVehduvWqmvWuLZZulRzyzKOHw9c+kPUihVWdASo/t//Fb3t7t1Wxg/YU/nTT1uMLamsLKvXWLTIMnGe7NnjuvkuX573u3/+09ZfcEHhxzh1yuo2qla17erVU/3mG+/TuGOHarNmtm+7dnkDyuHDqs8958ohuS81a1pGdNAgu4G/9prqpEmWm1i3znIcxdWrqFouzpkziY/X3Azy5s3eXwMxIJRvOTn2l+j86xs71uoZpk61wuZ69fL+dT70UOE1gcnJ9v2LLwb+GkLQd9/ZDR5Qfe+9wreZNs1VP1C/vur8+YFJ28iRmlus5bRhgyuXMXdu0ftv26bao4frJ3HnnarHjhW9z2+/qTZpormljKmphW+Xman60UdWzdWmjSuwerNcdpnqG29YW4f8Zs+2wAJYjmrLFmsEAKj26eP/YqpQkZamOmFC6R46nBgQIsGrr7r+svL/FSYmWu1k/kdKd9On27bnncemIw5vv23/JNHRdvN3ysqy+gD3iuODBwOXrtRU1xPy7NmWCeza1T4PGeLdMbKz7SfjzG00aGDtDNzl5Ni1btrkqtu4/PKSZSJzclT377c6lAkTLAc1dKjqzTfbM0jr1lZE5wy+ztLM/v1Vv/7aAsyrr7q+79fPboqq9m/ubCU2ebL3aQo32dmq8+ap3nGHauXKdr1lqQdiQIgU773nKshOSlJ95hnLk3vz+JSTo3rppbbvO+/4P61h4vHHNbeSd9UqK7JxLyIaPdp3LXdK4vnnNfdp/fXXXbG8pGX2mzbZTd55M46LsyARE5P3mQKw63bejH3t1CnViRPtad89ODiLtwCrp8n/b+289iZNii5q+/xz26Z5c9Xf/1714YetnuOHH0oezA8cUF25sqRXWHLbt1sAdTYhdi5XXqm6YEHpj8uAEEk2bSp9n4JJkzS3ELosedJyJCfH1Qy0Xj1rPeQsIvrhh+ClKz3dVV7vrBr68svSHSsry+oW3CuGnUtUlDVAGzDA+jcEwr591kekbVtLQ+XKnttMZGW5nmM81feMHet6TvK09OhhrbiKM3GiK3c2b17pr7Eoe/e6WpY7l0aN7Pq2bi378RkQyDtZWa5aw88/D3ZqQsaZM6o9e7r+OK+5JrBFRJ785z+uNA0YUPbjnT5txUHp6XbNwcj55Ld+vVVmF2XxYvs3qFQp7w0zO9tV3wJY0EtJUf3f/6za7Y47VDt3duVCYmNVX3rJ/gzyO37c+nC636Svu86315qTo/rxx65isNhY1dtus+IhXz6fMSCQ9956S3OLnCKlps4Lx45ZJeYrr4TGjVLVWua0aWM5F1/0sA5nzm45zpv02bN2wwes+Ovjjz3ve/So6l13uW70nTqp/vyz6/vFi13FNlWqqL78sgUfwLsWTlu3WuAZONDqOgrrNb5vX97mtH37Ft10uSwYEMh7p0+7WibNnBns1FAxTp8OXFFOKDtwwFWUM3Gi6u9+p7l1EPkryz2ZPt1VeV6hgvUkHzXKVafRoYOVyKpa5T2g+re/FX/cG2/Mm7OIj7eHi1mzLDfy3/+6WlDFx6t++KF/n8UYEKhkXnjBfh49ewY7JUReGzcu7423Th3rU1ISaWnW0c79OCLWqsy953pKiivgFNVkd8UKzS3+efZZ67/hfuy4ONf7Pn2KH6fKFxgQqGSOH7dOakDe9pZEISwry1UR3ayZ9VUorblzrUVSYqLn/iXOcSZfecXzca65xrYZOdK1buNG1aeecg0aUL269SQPVAmtp4DA4a/JsyefBEaPtrGnH38ceOop7yY7Jgqi7duBTz8F/vIXoF69sh3L+ezuafj1774D/vAHIDHRhhuPjs77/fz5QK9eNo/F9u1ArVoFj79lC1C3bmCnTOcUmlRyTz1lgUDVAkPXrvbrJQphTZvas0xZgwFg81cUNRfHddfZPFU7dpW5LMAAABDKSURBVABTpuT9TtX+fADg0UcLBgPn8Vu0CGwwKEpYBgQR6Sci76alpQU7KeVbhQrA88/bVF2NGwMrVtiUZe+9Z792oggXFQU88IC9Hzs273dTpgDLltnT//DhgU9babDIiLxz/LhNTDxxon3u3x+4+WbgzBng7Fl7db6vWBGoWdMee9xf69cvfBJkojB24gTQsCFw8iSwerXN9Z2dbbPvrV8PjBvnChqhglNokm98+ikwdKj9FZRUhQpAly42X+XVVwOdOrFOgsqFESMshzB4MPDRRzb96513Ak2aAL/+ClSqFOwU5sWAQL6zcyfwwgtAWpr90itVslyB8/XMGeDYMctVHDvmWvbsAXJyXMeJiwOuvBK45hpgwACgUaPgXRNRGfz2G9C8uT3fbN1qP+vt24EPPwTuuivYqSuIAYGC79gxq4+YM8eWzZvzfn/FFcDAgTarfIMGxR8vJ8fy5PPm2bJwIVC1qu1/yy3AZZeVfFZ7olLq3x/49lugVStgwwagZUvg558LtjwKBQwIFHp27wbmzgWmTgWmTQNOn3Z917Wr5Rpq1bIC2Zwce83OthzI8uXWpu/wYc/Hb9IEGDTIlqQkBgfyK2cTU6evvgJuuCF46SkKAwKFtvR04PvvgS++sOCQmendfg0a2F9hr15AcjJw8CDw+ee27Nvn2q5+fWuP2KhR3qV5c+CSSxgsqMxUrRFeSopVjy1bFro/KwYECh/p6ZZrmDMHOHfO8tzOJSoKiImxfHmvXnZDL+yvLicH+PFH4LPPgC+/BA4d8ny+5GTgxReBzp39dklldvw4MGoU0KGD1VxSSJo92/ocvPtuaP+cGBAocmVnW8+h3btdy5499rp4sd1sAeDGG4HnngMuvrhs51MFUlOtjmTLFiA21o4dE1O64+3dC1x7rRVIV6hgtZXe1LEQecCAQFSYY8eAl16yNoOZmZYL+dOfgKefLtlN97PPrMhr82ZbnEHGqVs3a7LbuHHJ0rdxI9CnD7Brl+WEVIFHHgH+9a+SHYfIDYeuICpMzZrWhHbrVuCee2zd+PFWFDV2bPE9slWBv/8duPVW4L//tcru48etSW379lahXb++5UQuvdSKr7y1dKkFkl27gMsvB6ZPt/XvvFMw4BD5QmEj3oXLwtFOyec2bVK96SbnmGY2d2ZGRuHbnj3rmpUlJkZ1zBjVhQttYH73YSsPH84788lf/mJTkhVlyhTXbOr9+rkmC3YOr/n88z65XIpM4PDXRCXwxRc2NRag2r59wTmqT52yqbmcU2hNn1708XJybDZ45zRbF1+sunatTb2WkaGammpTnm3bZvNiOidJvvvuvPM5zphh6+vVs5lxivLYYzbx80svFT5FF0UsBgSikkpJcc0vnZDgGhQ/NVW1SxdbX7u26k8/eX/MdetUW7Z05RaKWkaNKjhAfk6Oa4aVt9/2fJ6vv857rGbNVL/8klOikqoyIBCVTmqqau/e9qcSHa36z3+qtmplnxs1splOSurUKdV777ViJudUWjVq2NN806Y2SfL773vef+JE269588JnXt+9W7VWLdvm3ntd6QVUe/RQXbWq5Gn2JDvbis4orDAgEJXWuXOqf/973ifu1q3LPtdhVlbpntizsixwAFa0lT+tycn23bXX2vGzslTffNNyM875IO+6S3Xp0sIDircWLLDcTrVqqsOHq27fXvpjlcbSpRYcmespMQYEorL6/HOb6/DKKy3nEExvvKG5s7673xBHj9bcOoaDB/Puc+yY6sMP2+zx7pMODx5sgeX4ce/OffiwBZT8RVxRUao331yyIrTSOHFC9b77XOcdNcq/5yuHPAUE9kMgKomzZ61zWLDHJMjIsLGajhyx8aB69bKxErp2tY54M2cCvXsXvu/WrcBrr1lv8O3bXetjYoAePexYXbva+AtVq7q+V7WxnR991DreVaoEPPGE9ZN4/XXri3HunG17xRXA3Xdbz/K0tLxLerptU1gP9I4drRNfzZqFp33WLJsbc9cu2z4nx5YPPrD+I+QVT/0Qgv6UX5aFOQSKaM8+a0/IvXurpqW5ipEefti7/XNyVNevV33xRdXu3e0J3/2JPyZGtWNH1QcfVP34Y8sZOb+76irVzZvzHm/PHitaq1GjYO6hJEvFiqr9+1uuxdnk9/hx1SFDXNt06GAV9P/5jyuts2b59J+3PANzCETlzNGj1vP51Cl7sl+40EZ1Xbq0dDOyHD1qg/EsXmzjQK1bl3f+CsDmg/z3v4HbbvOcS0pPt4kA5s+3HEZ8fN4lLs62c45e61xOnbKBDefNc503Lg7o1w/44QcbwqNiRetF/uijrqFARo60nttxcZb2tm1Lfu3p6TabXyiOVe0HHLqCqDxyTtUFAFWq2ByOF13km2Onp1sx1JIl1gO7eXPg//6v8NnifWn/fhutduJEm8fb6fLLrWioZcu82+fkWE/xL76w4UZ++snmtCyMqhU3rV0LrFljr2vX2qRPV15pRW2hNr2ZHzAgEJVHu3cDzZpZ2f348cCQIcFOkW9t2QJ8/TVQrx5wxx2en+AzM23mPWcOYdEioHp1CxYpKTYx04IFtv7oUc/nu+MOYMIE7+qIVINXl3TunNW7RJVu9CEGBKLy6n//s+G9hw4NfmV3MKWmWmX25s32WreuFTUdO5Z3uzp1rGitXTtbkpKskr5HDyu2Gj3aKss9OXnSKsxXrrQitmbN/HtdgCu3tnixBbWffrKcW2mKx+A5IJRyPF4iChk33xzsFISG2rWtDqJLF7tZOjVuDPTsafNeJCdb66zCAufEiTYP5j/+AbRoYVOx5rd7t9VprFtnn0eNskEN/WHJEhsMcdEiK97Kzs77/Zo1pQ4InjCHQETlS0oK8N57NtpscjKQmOh9zunll63CunJly1106uT6btUqCwb791t9ys6dVnSTkmKz7hVl+3ZrLpuUZMOX16vnedvduy0Nn3/uWhcdbft26wZ0727Ngos6RjHY7JSIqDg5Oa7mreedp7prl62fPNk1+mxysnVMHDbMPg8YUPwxnaPUAnac4cNV9+3Lu93p06rPPecaVLFyZdVHHlGdPds64/kQ2FOZiMgLZ864hv9o1856f4vY57vusu9V7YbuDBLLl3s+3gcfaO4Aiddf7woMlSqp3n+/DYHy7beugRQB6/G9Y4ffLpEBgYjIW6mpqhde6LpBA6ovvFBw3KSRIzW3c2BhDhxQrVnTtvnkE1u3Zo3qjTe6juveIbB1a9W5c/17beo5IHDGNCKi/GrVsqE9atWyObG/+AJ47LGCdREjR1rz1lmzrGNgfsOHWyun3/0OuP12W9eunVUWp6QAAwdaKKhRw4YTWbvWhg4JElYqExF5cuSIte4pqgL3mWes93S3bhYUnEFj2jTguuusw+AvvwBNmxa+/65dFlRq1PB58j3hnMpERCWVkFB8a54RI6zJ6+LF1tMZsH4D991n75991nMwAKxZbACDQVEYEIiIyqJ6dStOAoAnn7QioCeftCf/Dh2ABx8MbvpKgAGBiKishg0D6te3vgpPPGH1AdHR1h8iJnz6/zIgEBGVVeXKlisAgDFjLJfw0EPWmSyMMCAQEfnC3Xdbr2jAxjd6+ulgpqZUGBCIiHyhYkXgzTdteO4JE6x1UZgJn8ItIqJQ17evLWGKOQQiIgLAgEBERA4MCEREBIABgYiIHBgQiIgIAAMCERE5MCAQEREABgQiInII6/kQROQwgJ2l3D0BwBEfJidc8LojC6878nhz7U1UtU7+lWEdEMpCRFYWNkFEecfrjiy87shTlmtnkREREQFgQCAiIodIDgjvBjsBQcLrjiy87shT6muP2DoEIiLKK5JzCERE5IYBgYiIAERgQBCRPiLyq4hsFZHHgp0efxKRD0TkkIj84raulojMFpEtjteawUyjP4hIIxGZLyIbRGS9iDzoWF+ur11EYkVkuYisc1z3M471TUVkmeM3/7mIVAx2Wv1BRKJFZI2ITHV8LvfXLSI7RORnEVkrIisd60r9O4+ogCAi0QDeBHAtgFYAbhWRVsFNlV99BKBPvnWPAZirqhcCmOv4XN6cA/CwqrYCcDmAYY7/5/J+7WcA9FLVSwG0A9BHRC4H8CKAV1W1OYBjAIYEMY3+9CCAjW6fI+W6e6pqO7e+B6X+nUdUQADQGcBWVf1NVc8C+AzA9UFOk9+o6kIAR/Otvh7ABMf7CQD6BzRRAaCq+1V1teP9SdhNogHK+bWrSXd8rOBYFEAvAF861pe76wYAEWkI4DoA4x2fBRFw3R6U+nceaQGhAYDdbp/3ONZFknqqut/x/gCAesFMjL+JSCKAJADLEAHX7ig2WQvgEIDZALYBOK6q5xyblNff/FgAIwHkOD7XRmRctwKYJSKrROQex7pS/85jfJ06Ch+qqiJSbtsdi0g1AF8BGK6qJ+yh0ZTXa1fVbADtRKQGgMkALg5ykvxORH4P4JCqrhKR5GCnJ8C6qepeEakLYLaIbHL/sqS/80jLIewF0Mjtc0PHukhyUETqA4Dj9VCQ0+MXIlIBFgw+VdWvHasj4toBQFWPA5gPoAuAGiLifPgrj7/5rgD+ICI7YMXAvQCMQ/m/bqjqXsfrIdgDQGeU4XceaQFhBYALHa0PKgK4BcCUIKcp0KYAGOx4PxjAt0FMi184yo/fB7BRVf/t9lW5vnYRqePIGUBEKgO4BlZ/Mh/ATY7Nyt11q+rjqtpQVRNhf9PzVPV2lPPrFpGqIhLnfA+gN4BfUIbfecT1VBaRvrDyxmgAH6jq6CAnyW9EZBKAZNhwuAcBPAXgGwBfAGgMGzp8oKrmr3gOayLSDcAiAD/DVab8BKweodxeu4i0hVUiRsMe9r5Q1WdFpBnsybkWgDUA/qiqZ4KXUv9xFBk9oqq/L+/X7bi+yY6PMQAmqupoEamNUv7OIy4gEBFR4SKtyIiIiDxgQCAiIgAMCERE5MCAQEREABgQiIjIgQGBIpaILHG8JorIbT4+9hOFnYsolLHZKUU897brJdgnxm2cnMK+T1fVar5IH1GgMIdAEUtEnCODjgHQ3TGm/AjHAHH/EpEVIpIiIn91bJ8sIotEZAqADY513zgGFlvvHFxMRMYAqOw43qfu5xLzLxH5xTGO/SC3Yy8QkS9FZJOIfOrocQ0RGSM2t0OKiLwcyH8jiiwc3I7IxovPzSE4buxpqtpJRCoB+FFEZjm2bQ/gElXd7vj8Z1U96hgqYoWIfKWqj4nI31S1XSHnugE2V8GlsB7kK0RkoeO7JACtAewD8COAriKyEcAAABc7Biqr4fOrJ3JgDoGooN4A7nQMI70MNpTyhY7vlrsFAwB4QETWAfgJNnDihShaNwCTVDVbVQ8C+AFAJ7dj71HVHABrASQCSAOQCeB9EbkBQEaZr47IAwYEooIEwP2OWajaqWpTVXXmEE7lbmR1D1cD6OKYpWwNgNgynNd9nJ1sAM56is6wiV5+D2BGGY5PVCQGBCLgJIA4t88zAdznGEIbItLCMZpkfvEAjqlqhohcDJuu0ynLuX8+iwAMctRT1AHQA8ByTwlzzOkQr6rTAIyAFTUR+QXrEIiAFADZjqKfj2Bj6ScCWO2o2D2MwqchnAHgXkc5/6+wYiOndwGkiMhqx1DMTpNhcxSsg812NVJVDzgCSmHiAHwrIrGwnMtDpbtEouKx2SkREQFgkRERETkwIBAREQAGBCIicmBAICIiAAwIRETkwIBAREQAGBCIiMjh/wH3IUHst0x/BgAAAABJRU5ErkJggg==\n" + }, + "metadata": { + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "The log-scale for the loss in $x$ nicely highlights that the SIP version does inherently better, and shows a significantly improved convergence for the NN training. This is purely caused by the better signal for the physics via the proxy loss. As shown in `loss_function()`, both variants use the same backprop-based update to change the weights of the NN. The improvements purely stem from the higher-order step in $y$ space computed by the inverse simulator.\n", + "\n", + "Just out of curiosity, we can also compare how the two versions compare in terms of difference in the output space $y$." + ], + "metadata": { + "id": "-ZIRzk7ZwZ2x" + } + }, + { + "cell_type": "code", + "source": [ + "fig = plt.figure().gca()\n", + "pltx = range(len(loss_adam_y)) \n", + "import scipy # for filtering\n", + "fig.plot(pltx, scipy.ndimage.filters.gaussian_filter1d(loss_adam_y,sigma=2) , lw=2, color='blue', label=\"GD\") \n", + "fig.plot(pltx, scipy.ndimage.filters.gaussian_filter1d(loss_sip_y ,sigma=2) , lw=2, color='red', label=\"SIP\")\n", + "plt.xlabel('iterations'); plt.ylabel('y loss'); plt.legend(); plt.yscale(\"log\")\n" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 300 + }, + "id": "KcJemrzNu6dm", + "outputId": "284fe910-faf9-4452-aa1b-b533e77a1818" + }, + "execution_count": 152, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3debyWc/7H8den076fNtp0aloUJXVCkxKRSGMXWWKMBjEzmCHMDMOYYRiMYYzsP4wYM4xCSdok0W5JC0ml5UhpU53q+/vjc5+lOqfOdp/r3Pf9fj4e1+Nez7k+d47zPtd3tRACIiIiAJWiLkBERCoOhYKIiORSKIiISC6FgoiI5FIoiIhIrspRF1AajRo1ChkZGVGXISKSUGbNmvVtCKFxQa8ldChkZGQwc+bMqMsQEUkoZrassNfUfCQiIrkUCiIikkuhICIiuRK6T0FEpKxlZ2ezYsUKtm3bFnUppVa9enVatGhBlSpVivw1CgURkXxWrFhBnTp1yMjIwMyiLqfEQgisW7eOFStW0Lp16yJ/nZqPRETy2bZtGw0bNkzoQAAwMxo2bFjsK56UDYXt26OuQEQqqkQPhBwl+RwpFwohwP33Q+fOsH591NWIiFQsKRcK2dkwahQsXgxDh8Lu3VFXJCKyrzVr1jBkyBDatGlD9+7d6dmzJ6+++iqTJk2iXr16HHnkkXTo0IE+ffowZsyYMjtvQoaCmQ0ys5Hff/99sb+2alV4+WVIT4fRo+G+++JQoIhIKYQQOOOMM+jTpw9ffvkls2bNYtSoUaxYsQKA3r17M2fOHBYuXMhDDz3ENddcw4QJE8rk3AkZCiGE0SGEYfXq1SvR12e0Cvz3rgUA3HILTJlSltWJiJTOu+++S9WqVbnyyitzn2vVqhXXXnvtPu/t2rUrv//973n44YfL5NwJGQqlsn07nHkmfW/ozn0/+5xdu+D882HNmqgLE5GKxiw+x4F8+umndOvWrch1duvWjc8//7wUnzRP6oVCtWpQty788APXf3QB/Y7dzqpVMGQI7NoVdXEiIvsaPnw4RxxxBD169Cjw9RBCmZ0r9UIB4OGHoU0bbN5cXut0C02awLvvwh/+EHVhIlKRhBCf40AOO+wwZs+enfv4kUceYcKECWRlZRX4/jlz5tCxY8cy+cypGQp168K//gWVK1N75P2MvW4cZvDHP8K4cVEXJyKp7oQTTmDbtm08+uijuc9t3bq1wPfOnz+fO++8k+HDh5fJuVMzFACOPhruuAOAIx8cyr2/WUsIcOGFEOvgFxGJhJnx2muvMXnyZFq3bs1RRx3F0KFDueeeewCYOnVq7pDU4cOH89BDD9GvX78yOXdqr310443w9tswaRLXz7+U8f3fYNzbxt13ewuTiEhUmjZtyqhRowp8rSTD8Ysqda8UANLS4LnnID0dG/sWT3b9OwDPPw+FXKmJiCS11A4FgBYt4IknAGj+4G+4sPN8vv/eJ7iJiKQahQLAWWfBsGGwYwd/X3cBNdjKyJFRFyUiUv4UCjkeeAAOPZT0bz7jxmoPMX06fPxx1EWJiJQvhUKOmjXhr38F4OrqTwJBVwsiknIUCvn17w9Nm9Lk+yX0YhrPPacOZxFJLQqF/CpXhksuAeDGxs+ow1lEInPXXXdx2GGH0aVLF7p27cqMGTPo27cvM2fOBCAjI4POnTvTpUsX+vfvz+rVq8vkvAqFvV12GQCnbHyJmmzhsccirkdEUs706dMZM2YMs2fPZv78+bzzzju0bNlyn/dNnDiR+fPnk5mZyZ/+9KcyObdCYW8dOkDPnlTZvpmLa/yHDz6A+fOjLkpEUsmqVato1KgR1apVA6BRo0Y0a9as0Pf36dOHJUuWlMm5FQoFiV0tXN/gaQAefzzKYkQkMhGtnd2/f3+WL19O+/btufrqq5k8efJ+3z9mzBg6d+5cJh9ZoVCQwYOhRg3ar5xEa75Uh7OIlKvatWsza9YsRo4cSePGjRk8eDDPPPPMPu87/vjj6dq1Kxs3buTmm28uk3Mn5NpHZjYIGNS2bdv4nKBuXTj7bHj+eW5t/iw/W/kHXn4ZLr00PqcTkQqqDPcpKK60tDT69u1L37596dy5M88+++w+75k4cSKNGjUq0/Mm5JVCabfjLJJYApy37VmM3epwFpFys3DhQhYvXpz7eO7cubRq1apczp2QoVAujj8eWrWizrplnFZrojqcRaTcbN68maFDh9KpUye6dOnCZ599xu23314u507I5qNyUakSDB0Kd9zBb5s/zehF/XjiCXjooagLE5Fk1717d95///19np80aVLu/a+++iou59aVwv7EmpAyl/2HunzPa69F2sQoIhJ3CoX9ad0a+val0vZtDKv7EsuXa5E8EUluCoUDic1Z+Hn1ZwAYMybCWkSkXIQkaRIoyedQKBzI2WdDnTq0XTudDnyuUBBJctWrV2fdunUJHwwhBNatW0f16tWL9XXqaD6QWrXgvPPgySe5vNIz3PTB3WRlQePGURcmIvHQokULVqxYQVZWVtSllFr16tVp0aJFsb7GEjkNMzMzQ86KgXE1aRIcfzzLanUkY8tnPPts7mKqIiIJx8xmhRAyC3pNzUdF0asX1KlDqy0LOIRlakISkaSlUCiKKlXgxBMBOJlxjBsHO3ZEXJOISBwoFIpqwAAAzqszlo0b4b33Iq5HRCQOFApFdfLJABy7/R0qk60mJBFJSgqFomrVCjp2pPqOTfRkukJBRJKSQqE4Yk1IZ1Yfy+LFsGhRxPWIiJQxhUJx5IRCjbGAZjeLSPJRKBRHnz5QowYZ6+dwEKsVCiKSdBQKxVG9OvTtC8AAe5upU2HDhmhLEhEpSwqF4oo1IV3UeCw7d8Lbb0dcj4hIGVIoFNcppwDw481vU4ldakISkaSiUCiutm2hTRtqbl1HJjN5803YtSvqokREyoZCobjMcpuQhqSPZd06mDEj4ppERMqIQqEkYqFwejUNTRWR5KJQKInjj4cqVWi19kMasE6hICJJQ6FQErVrQ+/e2O7dnFbtHT7+GL75JuqiRERKT6FQUvmGpgKMHx9lMSIiZUOhUFKxUPjxxrFA0HwFEUkKCoWSOvxwaNaMWhtX04X5jB8Pu3dHXZSISOkoFEoq39DU8+uNJSsL5s2LuCYRkVKqMKFgZm3M7EkzeyXqWoosFgpn1XgL0JIXIpL44hoKZvaUma01s0/2en6AmS00syVmNgIghPBlCOHyeNZT5k48ESpVol3W+9Rki0JBRBJevK8UngEG5H/CzNKAR4BTgE7ABWbWKc51xEd6OnTrRqVd2fTmPd57D7ZsibooEZGSi2sohBCmAN/t9fRRwJLYlcEOYBRwelG/p5kNM7OZZjYzKyurDKstoX79ALjw4Ans2AGTJ0dcj4hIKUTRp9AcWJ7v8QqguZk1NLN/Akea2c2FfXEIYWQIITOEkNm4ceN413pgsVA4wd4F1K8gIomtctQF5AghrAOujLqOYuvVC6pWpdnq2aTzHW+/3SDqikRESiyKK4WVQMt8j1vEnktMNWtCz55YCJxSfRILFsDy5Qf+MhGRiiiKUPgIaGdmrc2sKnA+8HoEdZSdfP0KoCUvRCRxxXtI6ovAdKCDma0ws8tDCDuBa4BxwALg5RDCp8X8voPMbOT3339f9kWXRCwUev7goTBuXJTFiIiUnIUQoq6hxDIzM8PMmTOjLgOys6FBA9i8measYFuD5qxdC2lpURcmIrIvM5sVQsgs6LUKM6M5oVWpAn36ADC40bt89x3Mnh1xTSIiJaBQKCuxJqRzG3gTkoamikgiUiiUlRNOAOCIdRPQUtoikqgSMhQqXEczQJcu0KgRNdetoL0t4f33YdOmqIsSESmehAyFEMLoEMKwevXqRV1KnkqVfO9m4PKMCezcCZMmRVuSiEhxJWQoVFixfoWBNTQ0VUQSk0KhLMX6FdqvnIixW/0KIpJwFAplqW1baNmSKt+v49ja81i8GJYsibooEZGiUyiUJbPcJqRhbX3V1Ndei7IgEZHiUSiUtVgo9MP7FV59NcpiRESKJyFDoUIOSc0R61c4ePEUalfdwfTpsHp1xDWJiBRRQoZChRySmqNZMzj0UGzLFq7q/iEhwOuJvQasiKSQhAyFCi/WhDS4ifcrqAlJRBKFQiEeYk1InddOoFIlmDABKmJLl4jI3hQK8dC3L5hRdeZ0+h2zhexsePPNqIsSETkwhUI8NGgAPXpAdjbDD30HUBOSiCQGhUK8DBoEwAmbRwPw1luwbVuUBYmIHFhChkKFHpKa47TTAKgz5Q2OPGI3mzd734KISEWWkKFQoYek5jjiCGjZElav5uqjZwFqQhKRii8hQyEhmOVeLQyyMYDPV9i1K8qiRET2T6EQT7FQaPLhaNq0gawsmDYt4ppERPZDoRBPJ5wANWtic+Zw6YkrAC2QJyIVm0IhnqpXh5NOAuCCum8A3q8QQpRFiYgUTqEQb7EmpB99NpqDDoKvvoJ586ItSUSkMAqFeBs4EAB7dwLnDNwKaBSSiFRcCoV4a9rUZzdv28ZlLbXHgohUbAkZCgkxeS2/WBNS1+WjqVMHPv5Y23SKSMV0wFAws1+aWV1zT5rZbDPrXx7FFSYhJq/lF1vyIu2tMZzxk90APP54lAWJiBSsKFcKPw0hbAT6A+nAxcDdca0q2XTtCs2bw6pV3HjSHAAeeww2bYq4LhGRvRQlFCx2eyrwXAjh03zPSVHkm918+NLRHHus76/w1FMR1yUispeihMIsM3sbD4VxZlYH2B3fspJQrAmJ0aO54Qa/++CDWvZCRCqWooTC5cAIoEcIYStQBbgsrlUloxNOgBo1YPZsBnVbSdu2PmdBI5FEpCIpSij0BBaGEDaY2UXAb4EEGfZTgdSoASeeCEDa2Df41a/86fvvj7AmEZG9FCUUHgW2mtkRwA3AF8D/xbWqZJWvCenSSyE9HaZP90NEpCIoSijsDCEE4HTg4RDCI0Cd+JaVpGKdzbzzDrVsK1de6Q91tSAiFUVRQmGTmd2MD0V9w8wq4f0KUlxNm0Jmpu/LOXYs11wDVarAf/8LS5dGXZyISNFCYTCwHZ+vsBpoAdwb16oOIOFmNOd3wQV++9hjNGsGQ4bA7t0+EklEJGoWirCOs5kdBPSIPfwwhLA2rlUVUWZmZpg5c2bUZRTPd9/5RLZt22DxYuZtaUvXrlCrFqxYAfXrR12giCQ7M5sVQsgs6LWiLHNxHvAhcC5wHjDDzM4p2xJTSIMGMHiw33/sMY44wgclbdkCI0dGW5qISFGaj27F5ygMDSFcAhwF/C6+ZSW5q6/226efhm3bciezPfQQZGdHV5aISFFCodJezUXrivh1UpgePaBbN1i3Dv79b04+GTp1gpUr4fnnoy5ORFJZUX65jzWzcWZ2qZldCrwBvBnfspKcGVx1ld9/9FHMYMQIf/jrX8OqVdGVJiKp7YChEEL4DTAS6BI7RoYQbop3YUnvggugXj2fuTZvHhddBAMGeD/0z36mfZxFJBpFagYKIfwnhHB97NBqPWWhVi245BK/H7taePJJn+X85pvwxBPRliciqanQUDCzTWa2sYBjk5ltLM8ik1bOlObnn4eNG2nWDP7xD3/quuvgyy+jK01EUlOhoRBCqBNCqFvAUSeEULc8i0xanTrBccf5eNRYD/P558N55/lTl16qpbVFpHxpFFHU8nU453Qk/OMfcPDBMHUqPPBAhLWJSMpRKETtzDPhoIPgk09g2jQAGjb0/gWAW2/1l0REyoNCIWpVq8Lll/v9Rx/NffrUU+GKK2DHDu+P3rEjovpEJKUUZZmLa80svTyKSVnDhvnchVdegays3Kf/+ldo3RrmzIFbbomwPhFJGUW5UjgI+MjMXjazAWZm8S7qQBJ6ldSCtGoFAwf65cAjj+Q+XacOPPsspKV5QNx9d4Q1ikhKKOoqqQb0x/dmzgReBp4MIXwR3/L2LyFXSS3MlCk+EqlGDVi4EFq2zH3phRfg4ou9H/rvf4drromwThFJeKVaJRUgtvPa6tixE0gHXjGzv5RZlamuTx8491z44Qf4zW/2eOnCC+Gxx/z+tdf6OnoiIvFQlD6FX5rZLOAvwDSgcwjhKqA7cHac60st997rVwovveRXDvlccUXetp0/+5m/RUSkrBXlSqEBcFYI4eQQwr9DCNkAIYTdwGlxrS7VtGoFN8WWlfrFL/aZuXbddXDHHb5T20UXwejREdQoIkmtKAvi3RZCWFbIawvKvqQUd+ONHg7z5sHjj+/z8m9/62/ZudNbm955J4IaRSRpaZ5CRVOjBtx3n9+/9VZfNjUfMx+FdPXVsH27D1p67rkI6hSRpKRQqIjOPhuOP94D4bbb9nnZLG8UUs7kthEjvFlJRKQ0FAoVkRn87W9QqZIvhPTxx/u8pVIlD4ZHHvF5DPfc4ytmbNoUQb0ikjQUChVV586+WN7u3d7pXMh8kquvhnHjfB+G11+HXr1gWYE9QCIiB6ZQqMjuuAMaNIBJk3wJjEL06wczZkCHDn5R0aNH7tp6IiLFolCoyBo0gD/+0e8PHw4rVxb61nbt4IMP4KSTfPmkvn3h97/3zmgRkaJSKFR0w4bl/aYfPBiyswt9a/36vpXn9df7kNU774Qjj/RtoEVEikKhUNGlpfmubM2aeZvQrbfu9+2VK/vieVOm+NXDggXez/CrX/lubiIi+6NQSARNmvi6FmlpvhTG668f8Et69/b5byNG+Eilv/0NDj9ck91EZP8UConi2GPz1s4eOhSWLj3gl9SoAX/+s3dCH3EEfPWVt0QNGgRz58a3XBFJTAqFRHLDDfCTn8CGDXDeeUXuRe7eHT76CO66C2rVgjFjvK/hvPO8eUlEJIdCIZGYwTPPQEYGzJzpPcpFVKWK79725Ze+sF61avDvf3uT0iWXwBeR7owhIhWFQiHRpKf7nIWqVX2286hRxfryJk18Ce4vvvC5cZUq+dpJHTr4ktxFaJUSkSSWkKGQdNtxFlf37vDgg37/pz8t0ZjT5s09UxYtgksv9QnTTz4J7dv73g0KB5HUlJChEEIYHUIYVq9evahLic6VV/qf9j/84Eulfvppib5N69a+k9vnn3sz0u7d8MQTCgeRVJWQoSB4/8Kjj8IZZ8D69XDyyfD11yX+du3awbPPFhwOw4fvs4K3iCQphUIiq1wZ/vUvn5SwciX07w/ffluqb1lQOPzjHx4OI0fusxmciCQZhUKiq1HDJ7N16QILF8Kpp8LmzaX+tjnhMH++b+2wbh38/Odw9NG+xpKIJCeFQjKoXx/GjvUOgo8+8k16duwok2992GEwYYJPqG7RAmbNgp494bLLYM2aMjmFiFQgCoVk0bSpb6zQpAm8/bbPei6jth4zn+j2+ec+16FqVZ8u0bGjh4WIJA+FQjJp1w7eegvq1PH5C0OH+nKpZaRWLZ8V/emn3q+9fj2cfz5ceKHfF5HEp1BINt26wRtv+G/wF16Aiy8u02AAaNvWs+ef/4SaNb2vu0sXb2YSkcSmUEhGvXt7U1LOFcOQIfvdh6EkzLzjee5c73xesQJOPNGX6P7hhzI9lYiUI4VCsurVy/sW6tb1RY4GDy6zzuf82rWD997znUMrV/YlujMzfdSSiCQehUIyO+YY30Chfn149VU455y47M9ZuTL87ne+2sahh8Jnn8FRR8Ejj/jyGSKSOBQKya5HD2/sT0+H0aPhrLNg27a4nCoz04esXnGFZ8811/jpNBtaJHEoFFJBt27w7rvQsKFv4nzmmXELhpo1febzyy9DvXrw2mu+wc/UqXE5nYiUMYVCqujaFSZOhEaNfKLb6afHtUf43HO9E/qYY7wTum9f+MMftEyGSEWnUEglnTt7MDRu7J3QcQ6GjAyYMsUnvIUAt9/u4bBsWdxOKSKlpFBINYcf7sHQpAmMH+/be27dGrfTVaniE97Gj4dmzXykUpcu8OKLcTuliJSCQiEVHXaYB8NBB/nopEGD4hoMAP36+TDVM86AjRt96sTFF0Oq7pMkUlEpFFJVp04waRIcfLB3Qg8cCFu2xPWUDRvCf//rHdE1a8Lzz3tXx7RpcT2tiBSDQiGVHXqoB0PTpn57+ulxG5WUw8yHrM6e7buKfvUV9Onj8xziMLdORIpJoZDqOnTIu2KYMMGXQy3jJTEKO+3778OIEd4J/cc/+oQ3zYQWiZZCQXxbtfHjoUEDn+B2ySXlMna0alX4859h8mRo0wbmzfMJcH/+c5mv4SciRaRQEHf44T5/IWcRvSuvLLc1Knr39kC46iq/SLnlFl+66fPPy+X0IpKPQkHy9OgBY8b4Fp9PPAHXX19uwVC7tu8FPW6c7/D24Ydw5JFw333l0polIjEKBdlTnz4+RKhKFXjwQbjttnI9ff/+8PHHvj/Qtm3wm994h/R775VrGSIpS6Eg+xowwGeXVaoEd94Jf/lLuZ6+fn3f7vONN7yv4eOPvYnpsssgK6tcSxFJOQoFKdjZZ8PTT/v9m26C++8v9xJOPRU++QR+//u8faE7dIDHHtMaSiLxolCQwl1yic80A7jhBm9OKmc1avhCep984k1L69d7H3iPHr53kMJBpGwpFGT/rrjC/zQHuO46eOihSMpo184HR738MjRvDnPm+JSKDh18r2htASpSNhQKcmDDhvlvXoBf/hIefjiSMsx8Se5Fi3xXt9at4YsvfChrq1Y+AU4b+oiUToUJBTOrZWbPmtnjZnZh1PXIXn7+cx8zCnDttf5bOSI1a8LVV3s4vPSS7yGUleVLZTRrBieeCPfc40tp7N4dWZkiCSmuoWBmT5nZWjP7ZK/nB5jZQjNbYmYjYk+fBbwSQrgC+Ek865ISuuqqvKuEa66J7IohR+XK3oQ0c6av0NG/v28DOmGCL5/RvbuvED54sHeNTJniezlotrRI4SzEcXKSmfUBNgP/F0I4PPZcGrAIOAlYAXwEXACcDrwVQphrZv8KIQw50PfPzMwMM2fOjFv9Uoi//x1+8Qu/f8klftVQu3a0NcVkZfmir+PH+/H11/u+Jy3NJ8hlZHizU9OmHh4FHZUrl/tHEIk7M5sVQsgs8LV4hkLs5BnAmHyh0BO4PYRwcuzxzbG3rgDWhxDGmNmoEML5hXy/YcAwgEMOOaT7Mm3jFY1nn/Urhx9+8F7gUaO8HacCCQGWLPFwmDYNli71K4Vvvina11eq5M1RLVv6ccghebcZGX7Urx/PTyASHxUtFM4BBoQQfhZ7fDFwNHAT8DCwDXgvhPDCgb63rhQi9tlncP75PrusalWf5PaLX3iPcAW2fTssX+7Ldi9bBmvWwNq1ex5r1vhVx4H+96hXLy8gMjI8MPIfTZp4uIhUJPsLhQpzcRxC2AJcFnUdUgydOsGMGfDrX3sn9K9+5Q36Tz0FjRpFXV2hqlWDtm392J8dO2DlSg+Qr7/Ou/36aw+Ur77ynePmzfOjIFWr+tVFu3a+fUXO0aGDb3xXwfNTUlAUobASaJnvcYvYc5KIatTwPoV+/eDyy33p7U6d4MIL/SriqKMS9jdf1ao+7LV164JfDwG+/TYvIL76as/gWL7cX//iCz/Gjt3z6+vV887w3r39OOYYqFUrvp9p504PsGnTYMMGH52199GiBZx5ps8HkdQTRfNRZbyjuR8eBh8BQ0IInxb3e6v5qIJZtgwuumjP1esyMnz4z+DBvvdmggZESW3d6v8sixb5UuD5jw0b9nxvWpp3y+SERK9e0Lhx6c6/fbuPzpoyxY9p02DTpgN/nZmf/7zzfMWTZs1KV4dULJH1KZjZi0BfoBGwBrgthPCkmZ0KPAikAU+FEO4q5vcdBAxq27btFYsXLy7jqqVUQvB1r0eN8unH+Xt127TxNpuDD/YhPwcfvO9Rr15KBEcIsHo1TJ8OU6f6MWfOvvMq2rXzX869esGxx3qzU2H/PDt3woIFHgKzZvnt3LkeDPn96EceOi1ben9H/sMMPvoI3nwz7+vM/NxDhsBPf+pXUJLYIu1ojiddKVRwu3f7VcNLL/lCRUVZ4rRaNW9szwmJDh3g6KP9aNEi/jVHaNMm+OADD4hp0/z+1q17vqdaNR/9W7OmH7Vq+W12tm9lWtByH4cd5iui9+njYVCUZqFNm7wl8OWXvdkrJyDatfO1EQcOTInsTloKBYnezp0+SmnVKv8TOf+xapUP91m9ev9tG82a5QXEscf6bRJPJMjOzmv/nzbN83XVqv1/TevWvqVpztGtW+mHzW7cCP/7H/zpT3m74Z18MjzwAHTsWLrvLdFQKEji2LLFA2LNGm96mjvXRzh9+KEP9cmvXj046STf/+Hkk5P+SiIEvxLYutX/mbZuzbsfgu+o2rBh/M6fne2DzG67zf9TpKXB8OFw++2Qnh6/80rZUyhI4tu923trZ8zw4913YeHCPd9z+OG+1sWPfww9e6p3NE6ysnyPi5Ej/T9Lw4Zw771w6aVqUkoUCgVJTkuXeoP32LE+P2LLlj1fP+QQD4djjvGmptatNZusDM2f74vmTprkj084wVdZP9D8D4le0oWCRh/JPnbs8Ib3iRN9SM+MGQX3T1Su7COfmjfPO9LTvfe2Tp09j0aN/L0NGuhP4EKEAC+84FttfPstVK/uzUs33ODbfEvFlHShkENXClKoXbt8fOb06X7Mng0rVsC6dcX/XlWrejg0a+a3rVv7rLPMTB/fqSsPvv3Wg+D//s8fd+kCjz/ucxel4lEoiOTYts07sFeu9OObb7zXdNOmfY+sLB/us3cHd3516+YFRI8e3oYSz97eCm78eN96Y+lSz8orr4Q77kjpf5IKSaEgUhpbt3o4fPON3y5c6DPDZs7cd8lVM//zOGdE1FFH+TCdFLJ1q49Iuv9+v2BLT/fHV12lJqWKQqEgEi+rVuVNH5461ScT7NiR93p6ug+bPe00n/HVoEF0tZazTz7xvoZ33vHHHTt6UAwYEG1dolAQKT+bN/twnJxRUV98kfdaWppPKT79dD8KW2kviYTgM6NvuMH3tgDPxvvu89ViJRpJFwoafSQJY8kSX0jo9ddh8uQ99wLt3BnOOMOPI49M6hFO27fDQw/BnXd6d42ZL7b329/69BIpX0kXCjl0pSAJZf16ePkCa5cAAAywSURBVOstXzPirbf2HDLbsmVeQPTunbSN72vW+JDVp57yGdLgy3T/7neei3Jg27fDp5/6KOr27Uv2PRQKIhXN9u0+p+J///Mj/6JG6elw6qneD3HyyUm5hsTy5b5R3+OP5y22N3Ag3HKLzzdM4oumYtmyxde/mjPHR1XPnu2BkJ0N117rV18loVAQqch27/aO6tde82PBgrzX0tJ83eyBAz0kOnZMqt+Yq1bBX/8Kjz6atyJs584wbJjv05SEebhf27f7tJrx472DfubMfZdTN/MrhCFDfLmRklAoiCSShQthzBh44w0f0ZS/H6JFC58P0b173lHcnXh27/YdfrKzPXT2PqpVK/fgycqCBx/0K4ecFdarV4dzz/WA6NUrqbIw165dPkrr3Xc9CCZP3nO59MqVfenzbt28ea1bNzjiCG86Kg2Fgkii2rAB3n7bA+LNN33q8N5atvQpxLVq+ezr/Edams/iXrs2b/XZtWv3DJq9NWrkCwuecoo3X5V2+7di2LHDW9NGjswbygq+rcY553iXS/fuiRsQ2dneBJSzE9577+27A9/hh/so5pNO8j0w4rFFq0JBJBns2uVXEbNm5c2NmDNn3514iqJuXb8i2LVrz2PnzrweYPDfvpmZPrngJz/x++Xkiy/gySfh6ad9q40cLVp4OJx5ZsXvk9+40ZuDpk2D99/3+3v/52rVCo47Dk480Y+mTeNfV9KFgoakisTkBMWiRd4gvWPHnkd2tk+YO+ggP5o08aN69YK/Xwj+/caO9RFSkyfvuZ/nwIFw993lOo40O9unfrz6ql9F5J9Enp7uv0j79/e/rFu1KreyCrR6tY8feO89D4L58/2fNL8OHfbcCS+KmpMuFHLoSkEkzrZu9d/Ib77pq91t2uSLGl16qS9qVJS9PctQTp/8q6/6sfeWGu3b5zW9HHdc6XedO5ANGzw3J0zw47PP9ny9ShXvB8jZZ/vHP/ZdZstECCVuR1MoiEjprV3rs8/++U9vZqpRw9exuPFG3wUvAosXewft+PHeWbtxY95rZt7V0rt33l/lpf2FvGbNntuj7j06qGZNP89xx3kI9Ojh/0wltn69T4D84ou8I+fxxRf7VVsJKBREpOwsXgy33gr//rc/btTIfzlddlmky4jv3Om7tuaExIcf7tk9Ar4BUPfuvsJImzZ5ty1b+l/1u3b5X//ffee/j7/7DpYt8/6AadP2XLUEfHTQ0UdDv35+HHOM9+8X2e7dfpLly/3fddGiPW/3t9T72WfDK68U42R5FAoiUvY++MCvEqZO9ce9e/vWax07RltXzA8/eDBMneojfd5/f9/N+XJUquTDPPNfaRSkVi0PgZzmoF69ChkeumuXT8JYvtyPr7/229Wr/YorK8tv163bdyLC3if80Y/8aNs27/6PfuRJVrlykf898lMoiEh8hAAvvujNSGvX+p/bI0b41OTCOrMjsnMnzJ3rcwOXLoUvv8y7Xbkyr4m+fn3vm09P9+Ogg3wF9F69vDmqwN/DK1Z4D/OkSZ5AS5d6MBRFeroPOWrXzjtF8t82bRqX8bcKBRGJr+++8zB4/HF/3K6dT1Pu1y/auopo+3bvU69Xr4gtYOvWwbhxHgQTJ+7brgSeJi1b+l7hLVv60ayZj/5q3NiPRo0iGVOrUBCR8vHee771Ws4wnEGD4OabfUGjRLd0qY+Jfe01/5z5rwTq1PHe7OOPh759fchutWqRlXogCgURKT87dviGCXfe6dufgv/CHDHCJ8GVZXPIxo3+V/v69d5DvH69H99/77+omzb1IUdNm/pf7kX9q3znTg+BhQthxgwPg48/znu9cmUfYtS/vwfBkUeWuH0/CkkXCpq8JpIA1qzxZTwfeSRvn+suXeCmm3xRo+I2m6xdmzebe9YsXy/i66+L9z0aNfJmm/r18zoNco4dOzwEFi70YZ97D12qU8eX/jj9dF/FNt6TIOIo6UIhh64URBLAxo2+mNH99+ctEV61qvc7dOzoW7B17OhH7do+BvTrr/e8XbTIe4P3Vr26t9Gnp+/5i75ePT/vqlU+4mfVKg+V/Y302VvLlj79+LDDPAz69q3QTULFoVAQkeht3w7PPQcPPLDv1N+iqF3bpwd375532769L/pXFLt2+VDQnOam/MeGDd7D3L69B0G7dvFZia6CUCiISMWyebM30yxY4Mfnn/vtDz/4aJ1Wrfa8bdPGx+ZHODkumewvFBKnZ0REkkft2nn7QUiFotgVEZFcCgUREcmlUBARkVwKBRERyaVQEBGRXAkZCmY2yMxGfp8zS1JERMpEQoZCCGF0CGFYvYh2exIRSVYJGQoiIhIfCT2j2cyygGUl/PJGwLdlWE6i0OdOPan62fW5C9cqhNC4oBcSOhRKw8xmFjbNO5npc6eeVP3s+twlo+YjERHJpVAQEZFcqRwKI6MuICL63KknVT+7PncJpGyfgoiI7CuVrxRERGQvCgUREcmVkqFgZgPMbKGZLTGzEVHXEy9m9pSZrTWzT/I918DMxpvZ4thtepQ1xoOZtTSziWb2mZl9ama/jD2f1J/dzKqb2YdmNi/2uf8Qe761mc2I/by/ZGZVo641HswszczmmNmY2OOk/9xm9pWZfWxmc81sZuy5Uv2cp1womFka8AhwCtAJuMDMOkVbVdw8AwzY67kRwIQQQjtgQuxxstkJ3BBC6AQcAwyP/TdO9s++HTghhHAE0BUYYGbHAPcAD4QQ2gLrgcsjrDGefgksyPc4VT738SGErvnmJpTq5zzlQgE4ClgSQvgyhLADGAWcHnFNcRFCmAJ8t9fTpwPPxu4/C5xRrkWVgxDCqhDC7Nj9TfgviuYk+WcPbnPsYZXYEYATgFdizyfd5wYwsxbAQOCJ2GMjBT53IUr1c56KodAcWJ7v8YrYc6nioBDCqtj91cBBURYTb2aWARwJzCAFPnusCWUusBYYD3wBbAgh7Iy9JVl/3h8EbgR2xx43JDU+dwDeNrNZZjYs9lypfs4rl2V1klhCCMHMknZMspnVBv4D/CqEsNH/eHTJ+tlDCLuArmZWH3gVODTikuLOzE4D1oYQZplZ36jrKWfHhhBWmlkTYLyZfZ7/xZL8nKfilcJKoGW+xy1iz6WKNWbWFCB2uzbieuLCzKrggfBCCOG/sadT4rMDhBA2ABOBnkB9M8v5AzAZf957AT8xs6/w5uATgL+R/J+bEMLK2O1a/I+Aoyjlz3kqhsJHQLvYyISqwPnA6xHXVJ5eB4bG7g8F/hdhLXERa09+ElgQQrg/30tJ/dnNrHHsCgEzqwGchPenTATOib0t6T53COHmEEKLEEIG/v/zuyGEC0nyz21mtcysTs59oD/wCaX8OU/JGc1mdireBpkGPBVCuCvikuLCzF4E+uJL6a4BbgNeA14GDsGXHT8vhLB3Z3RCM7NjganAx+S1Md+C9ysk7Wc3sy54x2Ia/gffyyGEO8ysDf4XdANgDnBRCGF7dJXGT6z56NchhNOS/XPHPt+rsYeVgX+FEO4ys4aU4uc8JUNBREQKlorNRyIiUgiFgoiI5FIoiIhILoWCiIjkUiiIiEguhYKkNDN7P3abYWZDyvh731LQuUQqMg1JFWHP8e3F+JrK+dbWKej1zSGE2mVRn0h50ZWCpDQzy1lV9G6gd2xd+utiC8vda2Yfmdl8M/t57P19zWyqmb0OfBZ77rXYgmSf5ixKZmZ3AzVi3++F/Ocyd6+ZfRJbC39wvu89ycxeMbPPzeyF2OxszOxu8/0h5pvZfeX5bySpRQviibgR5LtSiP1y/z6E0MPMqgHTzOzt2Hu7AYeHEJbGHv80hPBdbGmJj8zsPyGEEWZ2TQihawHnOgvf7+AIfLb5R2Y2JfbakcBhwDfANKCXmS0AzgQOjS1wVr/MP71IjK4URArWH7gktgz1DHwp5nax1z7MFwgAvzCzecAH+GKL7di/Y4EXQwi7QghrgMlAj3zfe0UIYTcwF8gAvge2AU+a2VnA1lJ/OpFCKBRECmbAtbEdrbqGEFqHEHKuFLbkvsn7Ik4EesZ2PJsDVC/FefOvzbMLyOm3OArfMOY0YGwpvr/IfikURNwmoE6+x+OAq2JLcGNm7WMrUe6tHrA+hLDVzA7Ft//MkZ3z9XuZCgyO9Vs0BvoAHxZWWGxfiHohhDeB6/BmJ5G4UJ+CiJsP7Io1Az2Dr8efAcyOdfZmUfC2hmOBK2Pt/gvxJqQcI4H5ZjY7tpRzjlfxfQ7m4Ttn3RhCWB0LlYLUAf5nZtXxK5jrS/YRRQ5MQ1JFRCSXmo9ERCSXQkFERHIpFEREJJdCQUREcikUREQkl0JBRERyKRRERCTX/wM5poKX1HNmbQAAAABJRU5ErkJggg==\n" + }, + "metadata": { + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "There's likewise an improvements for SIPs, but it is not as pronounced as in $x$ space. Luckily, the $x$ reconstruction is the primary target, and hence of higher importance for the inverse problem at hand.\n", + "\n", + "The differences in terms of $L^2$ error are also very obvious in direct comparison. The following cell plots a reconstruction from GD & Adam next to the SIP version, and the ground truth on the right. The difference is obvious: the SIP reconstruction is significantly sharper, and contains fewer halos than the GD version. This is a direct consequence of the undesirable behavior of GD applying diffusion to the backpropagated gradient, instead of working against it." + ], + "metadata": { + "id": "DeSQEpHB0Tip" + } + }, + { + "cell_type": "code", + "source": [ + "#todo plot outputs\n", + "\n", + "#vis.plot(generate_heat_example(batch(view_examples=2), spatial(x=64, y=64)));\n", + "#vis.plot(x_gd);\n", + "#print(format( tensor(x_gd.values,convert=True) ))\n", + "\n", + "#t1=tensor(x_gd.values,convert=True)\n", + "t2a=math.Tensor.numpy(x_gd.values ,\"batch,y,x\")\n", + "t2b=math.Tensor.numpy(x_sip.values,\"batch,y,x\")\n", + "t2c=math.Tensor.numpy(data.values,\"batch,y,x\")\n", + "t3=math.tensor( np.concatenate([ t2a[0],t2b[0],t2c[0] ],axis=0) , spatial(x=3*64, y=64))\n", + "print(format( t2a.shape ))\n", + "vis.plot(t3)\n", + "\n", + "#vis.plot( [math.tensor( t2a[0] , spatial(x=64, y=64)) , math.tensor( t2b[0] , spatial(x=64, y=64))] , row_dims=\"2\", col_dims=\"1\" )\n" + ], + "metadata": { + "id": "gNvf0FuxwfdF", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 759 + }, + "outputId": "775f10d9-a956-40d5-926e-63a9098c7c3e" + }, + "execution_count": 154, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "(128, 64, 64)\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAFgCAYAAAAmWu9SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOy9e7As2VXe+a3Kqjrn3Ee/pabVLSQBMg9jA3aHGAZikJARAmsQM0NgydgWDjl6wmFhZjxhS9gONIOJCdlEDEMMGkSH6FETYyRk2Rr1OARCA5I1DAjUAgF6ABKNhLppqdWv2/eeV1Vlrvnj1L357VWV++atyjrn1K3v13GiMyt37tyZuXfe2pXf+pa5O4QQQgghhBBiGXon3QAhhBBCCCHE+qOJhRBCCCGEEGJpNLEQQgghhBBCLI0mFkIIIYQQQoil0cRCCCGEEEIIsTT9k26AEEIIIYQQ1xPf9ZKz/sST5VJ1fPQPDt/n7i/vqEnHgiYWQgghhBBCdMgTT5b4nfd9+VJ1FHd8+raOmnNsaGIhhBBCCCFEhziACtVJN+PYUYyFEEIIIYQQYmn0xkIIIYQQQohOcZS+eW8sNLEQQgghhBCiQ46kUL7SY5jZfQBeAeAxd//6Odv/KYAfnK72AXwtgGe5+5Nm9lkAFwGUACbufncXbZIUSgghhBBCiI6plvyvBW8D0Oga5e4/6e7f6O7fCOBHAfwnd3+Sirxkur2TSQWgiYUQQgghhBBrh7t/CMCTVy14xKsBvH2FzQGgiYUQQgghhBCd4nCUvtxfV5jZGRy92fj3SROBXzWzj5rZPV0dSzEWQgghhBBCdEwHMRa3mdmDtH6vu9+7QD3/JYD/L8igvs3dHzGzZwN4v5n90fQNyFJoYiGEEEIIIUSHOIBy+YnF4x3FP7wKQQbl7o9M//+Ymb0bwIsALD2xkBRKCCGEEEKIjqngS/11gZndCODbAbyHPjtrZucvLwN4GYCPd3E8vbEQQgghhBBizTCztwN4MY4kUw8DeCOAAQC4+1umxf4rAL/q7ru06+0A3m1mwNFc4Bfd/Ve6aJMmFkIIIYQQQnSIA50GYM89hvurW5R5G45safmzhwB8wyrapImFEEIIIYQQHbN5ebc1sRBCCCGEEKJTHN5F8PbaoeBtIYQQQgghxNLojYUQQgghhBBd4kC5eS8sNLEQQgghhBCiSxyKsRBCCCGEEEIsjaGEnXQjjh3FWAghhBBCCCGWRm8shBBCCCGE6BAHUCnGQgghhBBCCLEsmyiF0sRCCCGEEEKIDnFs5sRCMRZCCCGEEEKIpdEbCyGEEEIIITqm8s17Y6GJhRBCCCGEEB2yqVIoTSyEEEIIIYToEIeh3MCIg807YyGEEEIIIUTn6I2FEEIIIYQQHaMYCyGEEEIIIcRSKMZCCCGEEEII0QGG0jcv4mDzzlgIIYQQQgjROXpjIYQQQgghRIc4gGoDf7/XxEIIIYQQQoiOUYyFEEIIIYQQYincFWMhhBBCCCGEEAuhNxZCCCGEEEJ0TCUplBBCCCGEEGIZjvJYbJ4wSBMLIYQQQgghOkUxFkIIIYQQQgixEHpjIYQQQgghRIcoj4UQQgghhBCiE0pX8LYQQgghhBBiCRy2kcHbm3fGQgghhBBCrDlmdp+ZPWZmH2/Y/mIzu2BmH5v+/Rhte7mZ/bGZfcbM3tBVm/TGQgghhBBCiI6pVu8K9TYAPwPgFzJl/l93fwV/YGYFgDcD+E4ADwP4iJk94O6fXLZBmlgIIYQQQgjRIceRx8LdP2Rmz19g1xcB+Iy7PwQAZvYOAK8EsPTEQlIoIYQQQgghOsRhKH25PwC3mdmD9HfPAk35FjP7fTP7ZTP7y9PP7gTweSrz8PSzpdEbCyGEEEIIITqmA7vZx9397iX2/10Az3P3S2b2PQD+LwAvXLZROfTGQgghhBBCiOsMd3/G3S9Nl98LYGBmtwF4BMBzqehd08+WRm8shBBCCCGE6BB3oFx98HYWM/syAF90dzezF+HohcITAJ4G8EIzewGOJhSvAvC3uzimJhZCCCGEEEJ0iqHCahPkmdnbAbwYR7EYDwN4I4ABALj7WwB8P4B/aGYTAPsAXuXuDmBiZq8D8D4ABYD73P0TXbRJEwshhBBCCCE6xLH6Nxbu/uqrbP8ZHNnRztv2XgDv7bpNirEQQgghhBBCLI3eWAghhBBCCNExq85jcRrRxEIIIYQQQogOcRgqX22MxWlEEwshhBBCCCE6ZhPfWGzeGQshhBBCCCE6R28shBBCCCGE6BAHUJ1wHouTQBMLIYQQQgghOsVQrjiPxWlEEwshhBBCCCE6ZFPfWGzeGQshhBBCCCE6Zy3eWAyGZ3175+aTbsZixLdgvkAdi+wTD+1USayPthlvq7yxXLI8s07LFi4Ar/fqea0XYY7bq8tlT7/lW0bLnP4i9a07VjXco3jP1+16tB0rTecV+mtSXdtr0bZcrq252xDHXps6c8da9B4vst+Cz7JT3w8XPa8No9y+Pn/LLA6qk27CRnJw+DRG491T/XSQFOqUsr1zM/7af/7DAADvpTcp+SKc+we3a+IXZm4Gb1pwYsFf/Kxst5PFLyNcx6Re7o3LtNxhvW60zUbjtByvjyfJNudtFdUxGKSN2hrW+5zdubI8uXEnKVZtF/Vy0XzPnbblyhV0jh6/PNK/dV4c30Mg3q8murDBjuNmcLG+Xz6oL0DvIO0b6K3ZQ7HtM4D6AF8b76fnm/SpcC34viTXd8GJRa+cP16B5ueBZSb/bZ8hsc/HvtJI5hnYfLBFfyU55f1w5ocWWj7lTT9OLnzlmWR9nVUiRnOJG/907+QassH8zsd+9qSbkMXdNlIKtRYTC3OgODwaxTYJvwzQPzgz/8gSC305i/+YNfSP+EW17T8k8YtwUmeffs3v85egUI6+BMQvD8kX12G9UozS4/b4i8Q2d4mttBxf+yrcB17lL+qDIinW9KWl2krLTWhiEb/s8X3uHdYH7o/SNnFfSb5k5fpJbB9/AV2zLwh8/6tB2nF4olXyGCrTa+gt1ZKW+cI4Mz5a7JOjqb5ra1NDuV56vk79stwKX8Abxm8chz2eJOReNnDfi09mmscn9cdzpGtT9TP1FfxMaR5f4ElMPBT3Fd5nJZPR0/1KYPYHr7q9bfvrJjI+v77XZnjhdPdJcTooN3BisXlnLIQQQgghhOictXhjIYQQQgghxLrgAKoN1EJqYiGEEEIIIUSn2EZKodZjYuF+RbPKsQfAnJiLy7tcyyRxSQ1s1HI7muMemGLSvK0kzW41qJcnwVWjHDYHjRrF4bLOuxoFHfmNLbsBy6hj21u60RQUE9Eb18vFbhoozuvVMI2/YDjuw2JQOl37RA8fdPRcf67ftDYKaNmfjjNmI/ZDvh7JcoyJadlGX+BXmdw+bQPbZ1ysuP4koDoEZQ8phsnmjzUAmJypy01CjAWXTWM2QjvYdIu9DGLwNsVRFON0Y/+gXh9eqAv2d6MRQ73NYhxU0qjMdaM+UA3rZ8NMkDeV47iMmTillrE0re85Vx2fecdoiJAcN/RDxVUIIY7yWGzes2DzplJCCCGEEEKIzlnpxMLMbjKzd5nZH5nZp8zsW8zsFjN7v5l9evr/NU1QIYQQQgghxHxK9Jb6W0dWLYX6aQC/4u7fb2ZDAGcA/HMAv+bubzKzNwB4A4DXX7Wm6Vt9C/qBJqnRtb2Jbvn+vGz4PGdLm5EB5JLRFftUbkKyjSAzmOyQTCpINSxRQjRb1jbKB2byYtTLUQpVkJKJ3/xNttM2leRgW4zq5eGlVLZRHDbbw7LHf0EWs8VMuVoW0rtUH8y309waye2LifqSgs2b0CSTaiut6oLYD5O+l2sI7RKLcZ0dyL/a4m3HZMayOSfPSXJX8CnO5LGgbUGRVw55meSPmZwGiewsjkMaAmWaJiYZ2xPqv/399BHe36sfUiw7jHAbY86MtP82y8kac75kLHBzZTvJ13JCqoOY4JNzBUkWJcRm4rCNlEKtbGJhZjcC+C8A/BAAuPsIwMjMXgngxdNi9wP4INpMLIQQQgghhFgTqjV967AMqzzjFwD4EoD/w8x+z8zeamZnAdzu7o9Oy3wBwO0rbIMQQgghhBDiGFjlxKIP4K8B+Fl3/yYAuziSPV3B3R0NOiQzu8fMHjSzB0fj3RU2UwghhBBCiO5wB0q3pf7WkVXGWDwM4GF3/+3p+rtwNLH4opnd4e6PmtkdAB6bt7O73wvgXgC44fyd9eQjJ4Fnve6M9ypXHnfMncYCtLRXTMpFi8aG84zxBmxRia20LGu4exQfEvsqa8UTDXjGvpbjI2aORZrtGEdQkYbdCypXBo3ypKR90m192pbo6GN8BMffsKZ8lArYjSw1473zjMY8pUGXvmbMWKW2jXVIdloweKT1tb72Y89YpfI696EYR8C2zyH+otzibZljUbdku9n+QWgkdcsqxi8M6zZOqI0xTqOkvtw/oPijw/DcoNikXpkGj7ENdMJM/FFv7iKqxexmV01TTFNbi+k1/TdeCHGCbGKMxcreWLj7FwB83sy+evrRSwF8EsADAF4z/ew1AN6zqjYIIYQQQghx3BwFb/eW+ltHVu0K9cMA/u3UEeohAH8fR5OZd5rZawF8DsAPrLgNQgghhBBCiBWz0omFu38MwN1zNr10JcdLpBSZgh2/mcq+6cpagNLH0Q6z4VxmstryoaKMhewxS1qO2YVZnsF2sDlb2nKc1lGQrKM45EaERlKdh+ebZVde1A2eyfJtdDJ0nXpFkHBQncUu2YHGbO28XjRn+c7SIN1ZNDNwJ2TkRKfSArMDyUzj9Y39ixOxs4ot04YqPi15XLItbSiX7Jc5xaSOMPbYBpeli2w3DaTSwOKgrnCwHzJ579f7FYdBanipXi4OWDaYjhu+VolcMfatgsdelBpiZax6rGVlUixZzTyzhRDXN2XnWvvTz6rfWAghhBBCCLFRODYzxkITCyGEEEIIITrF1jZOYhnWZ2IxvTdRwtGo6DhJqQcfO/apar7EKRdG33bCG/tvk1NNFdQ+1XD+crkdZAu8X1ATFSOSVhzwcnN7WYI1OZtuYxer6JDDTli9Mcss0jrYJaraqQ/W2x8n5ayqT6ayMCQaZAw5iVPufrV1oFk5SbbtZkla68zbTftEmrI6X0v9OXL1N+5Dy6Ff85iKmbcbq2v570g1aN42I4XqNWybkeDQ2CPntvJSWmyrx9Kl9KR7Y5IhjsiBrQwXh47VI+lPtZWOoVTW2TJ7ewd9YeEfCnk8ZLRrG/hDpBBCXJX1mVgIIYQQQgixJlSKsRBCCCGEEEIsw+UEeavEzO4D8AoAj7n718/Z/oMAXo+j99sXAfxDd//96bbPTj8rAUzcfZ7Z0jWjiYUQQgghhBAdcwwxFm8D8DMAfqFh+58B+HZ3f8rMvhtHiae/mba/xN0f77JBazexyNlBnkoaktgCqUY3a/9JOn8r0/PndQsZb7lOjreItpkcc8Hbyq30WNUOZ7lOt01K0ltTjEX/UjqoOOaiomzCo6A3T+wwQ7bxcshWmXTckKEbTRmE4zWkGItesKItt/tUbr301tk2ndQwWnXsU6Ptb86mud3FsMxYzsbH8DZO+J15+nKsEwBUA4phSKxtY1+uD8BdOVpM8/XojYMVLcc0cYxR+AeS47ZsTEFRM9czk1Gdy57Us71tn1xFTJAQ4rrlKEHeav/Nc/cPmdnzM9t/k1Y/DOCulTYIK8y8LYQQQgghhFiY28zsQfq7Z4m6Xgvgl2ndAfyqmX10yXoT1u6NhRBCCCGEEKedDoK3H+8i9sHMXoKjicW30cff5u6PmNmzAbzfzP7I3T+07LE0seiAmYzXbXUmC8hCohyDbVmtRDMtrW0Twis8p8zWxQ2jZFuPLsJkVGs1DnfSLta/yDqOerE8n57YIdWx9VSwGKbsvZNtlnS0vJ4zFsC8nNG7cBviPe/gbSfXcazWs13ThS1tF83g6rMWwA0W0DMFr7I+pZe6GSeZspkohWJZU7kd6iBZom9RH+2H/lqy1oqlkM3+tf2DdNvwmfpBklybYLfrJJPyYUYyyP0hjr3cM+u4aNtfJX26bhh/x4VkfWc4bijZzJPPnKmX//xMpmTKzmP1uPmyD+9e83HF+nBaEuSZ2V8F8FYA3+3uT1z+3N0fmf7/MTN7N4AXAVh6YiEplBBCCCGEEB1TeW+pv2Uxsy8H8B8A/F13/xP6/KyZnb+8DOBlAD6+9AGhNxZCCCGEEEKsHWb2dgAvxlEsxsMA3ghgAADu/hYAPwbgVgD/ux29gb5sK3s7gHdPP+sD+EV3/5Uu2qSJRQcs/KYrk0E3kRPQ8kxiZHI4KoK7S0VvdxMHl3gsVoKQsqIXZApJ4t1w0ls7tTRqm5YPhqkOZFzU63ZQayt8Kz3Y/nPonMu0m/ZIheV0jtUwSLdYqsHuNpkLEF23WP5wnFKlzo91Gt9NLigtyTswzc+8ncsobk4dO5cZParkfP5yTq5YkvtZdHRi+VO5k1bi5+q08n0aK/1BOm6MLs7osD5YWaRjaDSpT3Swm550uV13lsEz1IaYhZ6vLz9fQoZu3i/nrNY1bcdNJ0oFyaTWmhc/59PXvM+f3XjrleU/sDvb7/jY2Ws+llhT/FhcoV59le3/AMA/mPP5QwC+YRVt0sRCCCGEEEKIDnEo87YQQgghhBCiA05D8PZxcxoFEkIIIYQQQog1Q28sFiQ7CW1po+mJBjxYqnJcRRLbkGp5OWtuzKDLWanduL5wLE5+S1PNmP23N6o3lqMwJyW3vbNbdRDEoEg14Jd6pAHfqzXgFjJ5D284vLJ8uJtqUvu79bGLQ94SYkxY983xFtF6k+Mqgt1sem2ojhNLXX0dseJMxnzvot6eh0BuHGZjLpKs0ZmC3KX4uMG+lTPR+3baDwfbdYzFubN1+vpokzno1fvtjuoBfKG3k5QbU4zF+EL6z8Bkh8YHn1YMPyrmx1jE36vSeDE0Yi0vZ1uO9YfCVWeUF0KsHafFbva40cRCCCGEEEKIjtHEQgghhBBCCLEUjtW7Qp1GNLG4Bhr7R3wNzkqAXCLnlpmBU8lF2Jaxm3WSF3lirxnKJXaQXHd6rOT8g5yqovWCNA07g0labjvN2H2ZskzlE1vDer/D82lDJmfqsltPcwNDpdEe83KxIkg12A90xlKUKz3dD4i1e36twqLTG1eaaXndoo1sMlZyEh+WPDXIDgHAByRXDPbLAxoPN+7UUqizg3Q8DXt1ua2iXj4Yp4/68VYtQ5ycSa8TZ7PnZ0NvlLYpkT81WM8C4XlzGjNvr4KGZ48QQlzvaGIhhBBCCCFEx8huVgghhBBCCLEcrhgLIYQQQgghxJLIFUocwXrgLiwEZzTFHPfQo+XmKtiu0YN2l2MseqNgNxs9NqeMQx2JZS3ZUFoZG0XxHNupOLqq6nMZlbWo/Oww1YD3i1qM3u/XdcQYi4JsM20naLv7A7SBr02+IMeYpMdKrn2wB10lDbfu9LKKeIll8YblWKxojj/KxRz1yOm1or7h4anK+1mmTWwJXQT7ZR43Q4oJOtMP46sh2CPa0u5y/FUYTpPterka1idWHKTxUok9bIMt81EltE/bMblunMb+L4QQJ4AmFkIIIYQQQnSM3lh0jJl9FsBFHHl/TNz9bjO7BcAvAXg+gM8C+AF3f2qV7RBCCCGEEOK4kN3s6niJuz9O628A8Gvu/iYze8N0/fXH0I5uWXGm1UZ5RrSbZeVWUEH0JvNlPEWqikiPRXVUg/RgvkXypKDVGZKtrNNAYlkUkEqckrqDfW1J0qr+IEihOHsxJwnOyBGy0iK+l7GOar7eI1ffBj5HjhW+vjP3gbcl2euDxImTQff5/ofqWGoYZI3sUsxtKjPPBnKDnZFWGckQYzfkccMuIyyLAoABrXO5OO6MpFXxEqY2smjESnoe0DX0VbigNF1TSZCEEKcY38AvBDEC4Dh4JYD7p8v3A/i+E2iDEEIIIYQQokNWPbFwAL9qZh81s3umn93u7o9Ol78A4PZ5O5rZPWb2oJk9OBrvrriZQgghhBBCdEcFW+pvHVm1FOrb3P0RM3s2gPeb2R/xRnd3i5qaetu9AO4FgBvO33kq3ncnLiiL3u/EgajZ7SlZzzjJZOUerJKgBs9IMNi1hqQa7BAFACAnmeFWqqdi1xm+pTHjL8OvCL1slkJZL550w/KKe0knTk05CZ1kHcvD17ciqc4kSPBIQ+ekcYquRTGbfXKohszbMbd8Ra5LLLsqDkOfH9Xr5Tj9zWdMkkJenoT03T2fLzVscog72ims83OuIbs2EK7pgLWWzYcSQohNwZXHonvc/ZHp/x8zs3cDeBGAL5rZHe7+qJndAeCxVbZBCCGEEEKI40YxFh1iZmfN7PzlZQAvA/BxAA8AeM202GsAvGdVbRBCCCGEEEIcD6t8Y3E7gHfb0evzPoBfdPdfMbOPAHinmb0WwOcA/MAK2yCEEEIIIcQxI7vZTnH3hwB8w5zPnwDw0lUdd2k6sJH1TB1GAmbWc1thseDc+nozWnHaZRJiLKiO1Ho2ZN6ezC9n49AmirnohbiHnUEdY8EacM7IDaR6w+QVYRh8o1FdRzwWGnaLsSOdx1xwDEuMiUniPk7mQRJl9Nls7m3jObqO+1hxjEnSH+gexfijdDxkxsYhj9fm4/YPeFymQQaTLerztKkapnUUBxRjcZjWMRrVj+qDrXr50ngrKXeGYjgmVcsX0hnL3mwfomdRvL5CCCE2UwqlzNtCCCGEEEJ0iGMzg7dPIo+FEEIIIYQQ4jpDbywWhWUbOXlH3MRTObarDDaXSTZg/rwI9pKU/ZaX48G8P1/6AQA9Ul30SP5UBN/M3mFdXxUyZfcpsy8v740GSbmSZFJJQvGYeXvC9pUxa/ACsouWWbNnYMlTkvH7BH+FaNv3NoD4Y1BqCZ2TQlEm6/78exzXe8F6lp1eWVo1fCbVTNnZ+da21SD0+W2SGh4Eu9mD+lG9t1WPqWeKbTTBv5SV4UKlMsSwY9suRXa+yGY5bzfgNvCHPSEFnbie8c10kdfEQgghhBBCiI5Z1yR3y6CJhRBCCCGEEB3iUPC2uBosp6FZqCO++qddglTFG5xqYmZgI31Sklw39tH5iXYBAD3a6Ifz2wCkUqiCsv8Wh0mxxLVmdJhKnNiBZquobaaK4OjEEqpEThXOoyJXKKSHQsGGOazoyGX8zWRNt4y0KLlWxyg7StyNcq9SrydZVFP7F32XzPcuSA1tTBJCkhdWoQ0sk7IgNeyN5rcrft6ndrAMsQr9erJDY28/bcf4TN252SHqAlIp1KCYb11VtnWICmQlf3Q5WFrmw3CssmMtwCnUFlyTvHKD2Xl8EtZPph2f/6rzyfoHhy88mYYIcR2iiYUQQgghhBCdojwWQgghhBBCiA44hS9YV44mFkIIIYQQQnSMYixEexriLYDgoJezckysZ0PcA+m5KyO7ytBHOVPwTJwG2WMWGZ1z1a+7Adtm9oLdLOu+D/bTrnMwqdeHveYUxTzInHXf4fxBVVRleixy7EziKqIe3BKL2czPBm1jLMTJEGMvMveyKTZlpm9U3M+ps83c7ubYhF7DmIrHKkYUf7Bff14O06AgjmmahCFkZPU8GdfjIV4aHocFXYAyjC/LBAUkMV2Z0AweU07WszZp/xPdshnrc7ENbf89bxsfcS3fD2JsnThieGF09ULHQP/SmWT9wsWdE2qJEMthZvcBeAWAx9z96+dsNwA/DeB7AOwB+CF3/93pttcA+JfToj/h7vd30SYlyBNCCCGEEKJD3I9+TF3mrwVvA/DyzPbvBvDC6d89AH4WAMzsFgBvBPDNAF4E4I1mdvMSp3sFTSyEEEIIIYTomMptqb+r4e4fAvBkpsgrAfyCH/FhADeZ2R0AvgvA+939SXd/CsD7kZ+gtEZSqFXAGbWD3WrrTKMks0hkC71mqU48FEuj4jamInvIql8vcxZuACgOqO69VMaxezi8sjygzNuTMp278rlUJdvNBqnGpDkbsg/YspNkYsFuNrH2zVle0v3yfphrs6RhkSisOHXP2AOLFTIjcaNN1Dd6hxkZXxHqoDpZuhhlMFx/cUjHCpm8i0Oqb5LW0aNt1bhufNlLO9iIJYn91NozaVODnBBIr03ebpblT7Q8Tq+hF5yiPHo9c2pzrjsca4Gx17UFrCxlhRDXSgfB27eZ2YO0fq+733sN+98J4PO0/vD0s6bPl0YTCyGEEEIIITqmg+Dtx9397i7aclxICiWEEEIIIcT1xyMAnkvrd00/a/p8aTSxEEIIIYQQokMcywVud2RV+wCAv2dH/GcALrj7owDeB+BlZnbzNGj7ZdPPlkZSqC6IIrpkNadRbq4j0fOyjWyIAWANNMcbAECfbTRJDz1jI3tQi6yrYV1f/zBowA/qbcVueqy9va0ry1v9dnaz1aSuwybNMRYWqmMNeDWg5RkNPK1UmeAGrm+r37htxhJ3jZixv6zWWDC+bNzLTH20WMY+X3c+jkUCQn+jPlocpLEN5TbZOY9pHE5C/BHHVZUxsIhsmmls+DAtN5rUY5ljLGJoQ6+oG1wO0nOu+vxMyfT5cn5chcVnVJH5/ep6+WlrE7NgCSGuyqqfDGb2dgAvxlEsxsM4cnoaAIC7vwXAe3FkNfsZHNnN/v3ptifN7F8B+Mi0qh9391wQeGs0sRBCCCGEEKJLfPUJ8tz91VfZ7gD+UcO2+wDc13Wbrpffi4QQQgghhBAniN5Y5Oji9XbOXjTJDN1um43TClkKNWuHSfvR51amdXDm4d6I7GaDW2X/oG5Ifz891sGlWpO0t1UvF0V6rJIlHnQslFEKxe2LFpX1Ikuhom0mGuxmZyxlybKzCtmQl7aHlb3saslk5W7Kwj0Db4s/LiUZusPNJGkUS83isQqysC236/4VE9Qn4zzK/wraSGMll0E7TWodJF5U3yRKoWhMlcPm5wtY4jSuB6wN0jHkZ0jymJPgZaShbVnkx0HZyAohVsYGPl80sRBCCCGEEKJjVi2FOo1oYiGEEEIIIUTHbKKvgyYWkUV6QZRj5OprKps7Lm+K7lG8HCROLOsp9mlbUHSkUiiSaoxTyRBnBryopcwAACAASURBVOYs3ADQu1Tvt79dO0Rtn0ktqKoy6pUutz3UR1m/+bgAGl8teiZiaEbG0bSt7b3MlctlED4t5DIqbzKxbyWOUUFCSK5pPmjOLp1kot6i8TUKY7mavwwEuQ47RIVfw7hY2fKXMu+l7WBJYUmuUFWQEBYNYyBKDXl8ZaVQmWeg5EpCCLEeaGIhhBBCCCFEhzgkhRJCCCGEEEIsi2MxR4k1Z+V2s2ZWmNnvmdl/nK6/wMx+28w+Y2a/ZGbDVbdBCCGEEEKI48R9ub915DjeWPwIgE8BuGG6/q8B/JS7v8PM3gLgtQB+9hjacWzk9MBOKuikXMY2M6076KEzGn6u38kC0g7TgAbWPReHdYVF0IAXpCkf7KbtHdP6iDINj/oh7iNmFJ7SC5m3Oa4iZgpnG1xejnEaDGf/jZa9STbzljpvj2J8vn+LxlU0ZJSOP3hYLpt7S2YycV8vLHQNfe7izKZwzWwy/0Z7jLGgZc7KXW6nv+v0xrRPtKIdz+9fZcjePaHM26OiXo6v45P9cs8rLhZjJ2jdKDDDB/PjqI7qixng55c7zpiKmb7RxbGv0+ElhBBXY6VvLMzsLgB/E8Bbp+sG4DsAvGta5H4A37fKNgghhBBCCHHs+JJ/a8iq31j8rwD+GYDz0/VbATzt7pd/tnsYwJ3zdjSzewDcAwBbWzeuuJlCCCGEEEJ0hSl4u0vM7BUAHnP3j5rZi691f3e/F8C9AHDD+TvXdN6Wj9tpnRk4A0ujZmQGtK0ieUIRpFCgrNS98XzrWQAoSI7R300bzNIozi5c9gdJOfQTbcmVxWivyfKn/n7YRpIRlmvFTMZJVvJ+c/bfxA4zkwEdmezKiTTqNMqMYpNOYRMbbXpz71VPiZ0vWzZXW+ljNel7JJ/qTdJO1N/ncZjeoP4B2b6ShKraS491SNaxVdV8kydsK30YJFksL6QmVv1gbTukY3P2+iCFapKMLUon/043SOaEEKJTNvDxsso3Ft8K4HvN7HsAbOMoxuKnAdxkZv3pW4u7ADyywjYIIYQQQgghjoGVxVi4+4+6+13u/nwArwLw6+7+gwA+AOD7p8VeA+A9q2qDEEIIIYQQx44fGWcs87eOnEQei9cDeIeZ/QSA3wPw81fdw+yKI0t0RTpWTvnr8xmJT0N25Wo7ve09cqph2UL/INUWOUkhoiyivEDLQ9Z4pbKIaptcsRqcboAghdqL7lS0zFKocdQnNaXobnedroU0M3KzG9HSdYf6uU+e6DPoOMdG1/KnRLo2o3FrhvtNxf06jJsGl6T+7iRdP1P/zjO4mN7MajB/7FUHad0luasdsiQrZNc22q+/lx6LpYZMzF5fDckJarv5nxIjqWXMSt5WDrpQ324rSQzlGp37hBDiWtnAZ8ixTCzc/YMAPjhdfgjAi47juEIIIYQQQpwM6/nWYRlWniBPCCGEEEIIcf1zElIoIYQQQgghrm8khRKL6nKXpW1mYA+v1VrHnHAG3WF4UXVAy6SHtlEqZi8OKP5ikLZjeJHsJkkDbp4eq6Ts3UZ68Jhdmy1mo93sYI+yg1N90b6Ts4hz/Em0A02I17MhdqLt280ZjXau3zTdy5b75DJyx6pjPM6p5lpiKrqO78iFX7D9cNaKmJaTDPBpLEb/Ur2+FcZXsh9lr5/spMfqU+xEucVjMi3HcUoxxqKg8cbxETPn1RCbZOXyQTDZmIpjtHNe2BL8lMfjiZTJuDlbvBBLsYHDXxMLIYQQQgghusRxwq4qJ4NiLIQQQgghhBBLs/5vLPg10ymZGLZ+jZ95RZ68gs/oMVgaFWVRLHdxUidUg3Q+aWQVWezXXpO9aJt5SJm3g/WkJ/IEzi6clpuc4fOvF4sghRpQZu/Bfiqt6NN6j+RaxTgtx1KurKUsK5zayjgy9Xlb6VJLiYTnpv9NUq3ATN/oOBvyibGgzKRR4hLvXU7+xknkeTxEmVnVUEfoQ2w/OzNG6Xb19+tt5XY4FCk62KI2luP6WBYFAP2Dur0FWc/aJOrpmvp5LEdbckOj7XOzi0zsTVnexUZw68dDH/348BiPrg63SWyiEnL9JxZCCCGEEEKcNjSxEEIIIYQQQiyNYiyEEEIIIYQQ4tq5vt5YZOItcpPGa7IRnHfYRa0RO7EebWdFy5p6i/ERZD/r497cfQCgN6KYi0zbe1RHcRhiLLbpWDStLcbp+bLOu7+bxnokFpjcxDLEn/TpXHL2qnSd2NYzRy4+IhenwfVnYzFIfz8TH8JNzNiBZmMu2sZYNMUELdqvmXip11h6nFjPhvPivlJx/4qhCFRu+Mw43YjBlaU+2T5XwZaWrZ6530S72ZL2s3DdixHFN5EFbi/06ySGi5sR7n9uW2MfLZr7UHbcLOAaar3QhnJ+uWtiE4XVa8S5z+9fvZAQHbDs98t15PqaWAghhBBCCHHSOBRjIYQQQgghhFgW28gYi/WbWKyDXKKtNSITz4vqcGTkLhn72kSeUbJFazj0FmXrPVtLLoq9SVIuqeMw3cb0yPa1OAgZuvvzM3THrNks1ekFiVNVsNyj/rzcTnUQXGcihYrWq7weJVMsQ6qa7wNnUeblGdlGPyOFom0l3ROPshC2SqV+4+GnkeKA7Es9Heo2qre1zt5+jBmPs3QgM2n9erptBuXEUjXKfebvMnNf+RkQjjWgschyqv5uGPO9+WMj2tdOzrBMMG1Gj2SJxQH7VKflEomeN8iigFTWlHt+Z/pXVv6U1NGuWFJ3lJDyY6RsPq/0mdJtm4QQYl1R8LYQQgghhBBd40v+tcDMXm5mf2xmnzGzN8zZ/lNm9rHp35+Y2dO0raRtDyxxpldYvzcWQgghhBBCnHZWHGNhZgWANwP4TgAPA/iImT3g7p+80gT3/57K/zCAb6Iq9t39G7ts0/pNLKIcoSkp9Uy5lu+j2zrfLFI3gjxpAQlKlLukWYPjwWgx40bD0iKWLTjJooBUWjPTRHadIleVXnSI6ZGjDbtRRfUItWMSZBzVkCRUTbINpJIslnHNOCLl7nlDn7KQldwOyMWnomMhwK5bwYHKSa7Uo/OvQpvKHZJJUcLYKCcbPLFb17Gd3ku0zTC+Sto24bQ67OTcjprKMbns9fGc2f2LXIyinIrHEcufyu20r3Ff6QVHtuKQZYjtblJWWsb9MufOxpUEZybjZ+CiGbqbTqWL/pV59gohNpjVPwteBOAz7v4QAJjZOwC8EsAnG8q/GsAbV9kgSaGEEEIIIYToEsfRLyHL/F2dOwF8ntYfnn42g5k9D8ALAPw6fbxtZg+a2YfN7PsWPNOE9XtjIYQQQgghxPXPbWb2IK3f6+73LljXqwC8y935nfDz3P0RM/sKAL9uZn/o7n+6cGuhiYUQQgghhBCd00GCvMfd/e7M9kcAPJfW75p+No9XAfhH/IG7PzL9/0Nm9kEcxV9s1sTiWuxWW9exANk3VBzO0IVF56J2oHxp+s3t4OuRZslNy7EtrY1CPENDJueZ69Tni0PWs8MQb0DtrULbixHb2fJyGgPCMRbIZZpmnXqwtrWKJvZsqTkIKX4phiGxnt0/TMtVdSyGD0PcQ6++Bj06FwttKrcoToMsP4uDIEx/9LF621b6ZtTGdf3cR2MS4sZ+vmgM0wmxkuyny55z7JI8BKrmDNU2mR9vAQAVZ5unTRw3AcQYi+aM2k1tiPWnVswxiCuTAT6JA6M4ihh/xDa6mazcPFZsHGOpuFzmecDHytxjfuZFO18hhABwHDEWHwHwQjN7AY4mFK8C8LdjITP7GgA3A/gt+uxmAHvufmhmtwH4VgD/ZtkGrd3EQgghhBBCiE3H3Sdm9joA7wNQALjP3T9hZj8O4EF3v2wh+yoA73BPfvn5WgA/Z2YVjn7aehO7SS3KVScWU2uq/9Pdn1r2YEIIIYQQQohucPf3Anhv+OzHwvr/OGe/3wTwV7puT5s3FrfjyBf3dwHcB+B9YcazetyvvF7PZj9NNoRsqrkWL3s61/AWPM3y3K4NbWUcSSZcpJKBxuMC6B2ShCYjLeuNOLt0kE+wTIiP2wuShoZ9YnZtliT190I72NqW2hElHclxWarQixovkk9Ea9dBPUTKbV5utsBlu93BxVSe1d+tpVC2P062GdnUOu/WT4/V368PwBnQo41ueeGZur6d5yfbes/QSpI1GiknZfV6nNm1W1fYbO2a3a0pY3tUO2XkPqnsqPlYydiIYyqpr/l50GS/PHM9WZ6VyV7fKLVEKiFi+VOUKvEzyhIb6WD7nDlnb7Cize6TyLNS+WNy7MFW2l7KbF9tbbYwYHRz7YldBsnrzhcPjrs5QhwrK5HhnnKu+pXY3f8lgBcC+HkAPwTg02b2P5vZV664bUIIIYQQQqwnq7ebPXW0+q19+obiC9O/CY4CQN5lZo1BHma2bWa/Y2a/b2afMLP/afr5C8zst6epx3/JzIZNdQghhBBCCLF2eAd/a0ibGIsfAfD3ADwO4K0A/qm7j82sB+DTAP5Zw66HAL7D3S+Z2QDAb5jZLwP4JwB+yt3fYWZvAfBaAD971ZZefvsd5Aj8ypmlBFV8bZ3Lyt2Wllm5WztBJVKCBZuUOVSTNCgrJ8tcG3ZCiq5IXKeTdCc6OqXyrExb+VJP2rnWVP2WmrQohSJnnXIrPa/Jdr1e7tB5BVOoJDs6NWOyk86b+/uUXXsU5BOcoZgXw2nx9e2xU1e4LgXdy4Nnp8caPEqyK5bdZPou1x9ld52wSvlTru62WZ2zB848D1h2k9zXqAui5di/+FB8n6OMh6VLaDeuZw9A1yP3L1tDn2+Up2KOmxq1q9gbXVnuBZkgxuySRheqysgf++HfAJZlFplnBV/DSeb8+dDx3yVq/6ZLoYQQm0Wbb2K3APiv3f273P3fufsYANy9AvCKpp38iEvT1cH0zwF8B4B3TT+/H0Anmf6EEEIIIYQ4NeiNxSzu/sbMtk/l9jWzAsBHAXwVgDfjKOnG0+5XQlMbU48LIYQQQgixrmxi8PZK39FO04Z/o5ndBODdAL6m7b5mdg+AewBga+vG1TRQCCGEEEKIVaCJxWpw96fN7AMAvgXATWbWn761aEw97u73ArgXAG644S6/nIk52pwmWtYhnc6wOQbgRGeQGavIhIZ4jqwGPKej5niO6HPJ+mBeHKdWjg62hgyxEwOOP6BYl5CRljNqJ7rsYEPI2bWjXs9GIcN0Exz3wXr20HaOx8nZyHJ7Z84/ybZeL5dn0nKjc/WxisNwziQrZwvMGGNRsrWt1/VVoU03D+r4jr1npePhPK9w7EQmq3HCKcm8vbCNdNdxFcn9bx5fVWLfGmxZc9mriTQ2o13jLT53ssfilN005uOh+Fyo/pg1m2OfioPUfjmJkeA6QryU0T9Vzgnrc/c4Wl1zjFguyzdnL0/GRhiItC3GZvVPyqZZCCFOmGvIwHBtmNmzpm8qYGY7AL4TwKcAfADA90+LvQbAe1bVBiGEEEIIIU4ExVh0yh0A7p/GWfQAvNPd/6OZfRLAO8zsJwD8Ho7yYwghhBBCCHFdYK4Yi05x9z8A8E1zPn8IwIuuqS6rM3YORtGWlLI3k/VseXaQlmuQ+8Q6siwg98hJGjqxwE0qbM4MnEgLwqF6SSZbsiHtR1kQyW6CjWyaQbf5WleD+oOSlnvR1tGaLWsHLOvi/WImY9qvIhkDtwEAJmdICrXVfI/ZYnbGRjeRSXHdQQpFGqRemW4r9uvl/j7ZcI6SYijJOZblT+VOWu7Ws/UHhzeF9pIVJ9+73iRkMqZrXZFcrZcZh63HScs+v4oHcyLryWSbzx2b62C5z4ycjO1XObt2uIStXx+zhDCMm2jN3Ngmtn2O9zI5Vua+0rVK5Fkzds714iQ8l7m9iRRsEq4GX7dc/2LL8UF8ftXrMQN00iZuB12aaImd9KEZ6+AN/DYhhJhlTZPcLcPKpFBCCCGEEEKIzUGZe4QQQgghhOiaDXx5qYmFEEIIIYQQHaMYi9OK1XrZKFdzthRkm8BwM1n37kGMz9aeK2eBWIqcVS5rm2csUFl/T5riImiqnbT+5Vatga6G0TaS9cu5GAMqF3pYUi5pOxrpBYdKtnYsynqjB001nzPHW7DlKwAc3sjnH61CaZHtMIOIkNf5nMfn03KTc9XccgDQ360P1t+rlwe74Vis7SfJ+vh80NufPVsf9yxSyGKXLXWxj1bEvha1/idG0/jK6fI5ZmfGeZXiCKJmvz8/Dihq+yc79bbJNsdipIdK4oxiO7gfUhhMjE3i2JzBbj02it1xUo7PswqxVD2O0+DljC1ruVUvH96SxlGMzjZfe26H5Vyk2cK5dlGeGUM5990Zy92G43JMU/+gbmB/v/mZ1xuFsTcODy0hxGZySv5pPE4UYyGEEEIIIYRYmvV4YyGEEEIIIcS6ILvZ04sb4NOWViGjNsufOLPqrN1fs2Qo+wp+lXhL6QNvC+fFZ+LBAnWyU1+PyZn6Oo2DjoclAmMqV27PbfXRsaJiiE+FZUEz8qz59Q2faT5WhM/FSraKDVIokp2MSY6yf1tabnRjvTy+IdVLVDusGeENoQ81PDxmpHtna4lEbzvteJMbqb17dF5PpRetN5qv95icTRvh587UdeyEbdu1XIWvW5S/sa0s23C2TPg8y2nJSJyxmGUSuV6wFE0lafV1O7g5vV8Ht7BMCnOXj9abpYE9UjIVB/Vyfy8tN7w4/x5ZsEpN72V6Xiw1TDJPh/Nnm+aDW+p/SnbvaB5f5VbohwOqP7FvRVqu73OX0Y+2x1TJJG1vjzLd9w5JdhgkTv1LtJxIEkN9JKHdejq0Q1KolfHMC3auXuiE2XmifrYPnhllSl47j3x7qmvdf85JfYFZjO3HMrrnljznN6YPwWjzfBo5Jf/kHSdrMbEQQgghhBBirdjAiYViLIQQQgghhBBLsyZvLOxKttUZpyKSRjm5lsSM11XOFYa3JVKjmA263Wu3XLbtheBMsKHumL2XKbfZCYmcaXbiNayXx+do/2HLrOEAigPK5EtvZnPuSVxH/yAtZyQ16oU3pyXJqwqSUrD0CQDGlPV6/1aSqjw7bfzo1rrBZ56VWjA9+4ZaF7FV1PKGwzIdOmOWZJEc43DcXK4XLuLOsNa78H5Pb59LytklcsXaJxnTTpBxnalTdM/ch2Fd/4T6ySDI6WzSss+3KrUCWo61meSn7ASVkfsgZqxuYHQDuSLdnO4zvoGWybmrDPfLt9h2LK3fxiTjobE2uBSdmmjclJwpPe2H/T0epOGZwm5P7KwWrg0/Xw5uqcvt357WV95xeGX5/A2p7diwX7djZ1D3/6KXXhseK33SbloYQ33aj8caAOyO6wfd03u1nGbvmVTzOXq61qgNn5zvLAcARm56w0vhQVxlHsxiKXbvTPv8wTfuNZQ8PiZPD5P15z+wumMd3pr2rbf9zZ9b3cE64C2PviRZ/60/+sprrmPwpcHVC51SNjHGQm8shBBCCCGEEEuzJm8shBBCCCGEWCP0xkIIIYQQQgghrp31eGNhtf3iTMbnQYN1WbBXTH1Z001JZutF2hfltJnpmmVCOBLYDZKz/4bst8jpw7k62hSzQbPum+1WfTucWEGNKoPemLT+/V22gw0NYZdPOpXhhWD5uFcfuzcOFrvJ9ag/n4Ss2YnVLR83dJnihtoO8Dk3pb63L7zhS1eWn7f9xJXlM73UQrCkA+yVdWzDXpVqb58en6Hl1Dbx0rje74ZhrUV/5OxNSbnPf+nmK8vjC/U+fB4AUJ5Lj80krpzUjBin0t9nLT5tyA2U2P9PSm7eMiaq7T4WrJ6dbFlLei5Nghvm6AaKq7ihjtMZhPvVKzh2IK2jpPE2oXiJw2HUHlMMz5gsVQ+CZfF4fowJkNpx8zMlZqVna2p+pkxuSq1W73z201eWv/qmx5Jtd2xfuLK8TZ66g4wH+JgGcO8afg68UNY35tGDOvDls2dvTco9Oqz9cQ8LilMq0ms4oHCsmWdvuV4WoOuM9TY7nuWsdWtnKzpEeSyEEEIIIYQQnaCJhRBCCCGEEGJpNLE4vVyWtVh4888WgDG77CpJMk3nFBctmzRjI0taCJY/zbxy58zAMRt2w7FjuXKbjn1jLUfY3hkn5YaDWuIQx8r+Xi0ZGG/VN8miBGPE51XXUg2DtIra3psEKRSvN9jXAkAxIrkHSUmCignj/XoYPLmX6lge36qznN5C2ocohXpW/+KV5bPDx68sl8Hn9cAp43W4EXtVfQ2/OK7lGF++81RSrqL9HkEtk7rlxtQqd3yulkzNyP+oT7G8sAz3oUmSd00Py9OSbZtozDydkzGOgj1srWpLLJujtW9Fts2DG2vr1XNnDpNybJ0ah/mErGMpGTRGO+n44qz047Mkn9pOK+zTPe+lwzy1mC24XHofe6R4SrryML1Ot+zUdqBfeeZLybav3n70yvL5Xi3/u6lILUS3SRrF/f/A03/CeHztepAhljSW+7egid1RXcfTLEE7SG1pe2TFXEaJ7kSZt4UQm8naTCyEEEIIIYRYBwyKsRBCCCGEEEJ0gSYWpxNnV6ggH6lI0tE/JCeOoCVgaVEvZkVNLFgW6AUzUo+MNmoBWUiS8TXjQNUrg2QokXtkmscKlwll6w1uG5wZelikricDWr/YqyUD434qR6j25rt4VaEnsnNTlGD0d2uZwfh8vWOUTDG8jbN6RzgTMADsTer2P7R725Xl4lx6bQY2X/owDhZUI2835M4UtUzmC4c3JtvuPFs76XAG4Ru30vTlezssoUvr5+zKFTUx3ocEPuWcUXXHisQZiV/OWW0BIyiu34Mdk03InWySOTHaRIq2ozq36z7V79cFi17a+JhtOqmes9RTHx0PQrZ1kl2xi9OMYxrJncybHfSMnim55wvLojBJOwc/swe9dJzcWtSZ7Z/Xrx3Zzofn9430TDn0+jl04Gmfv1DVcqqnq2ZXtIN+LXe6ZZiOr/Nbtczx4qA+7mQrvU6Ts5ShPFxfG6xvpmAhREcckyuUmb0cwE/jyBbwre7+prD9hwD8JIBHph/9jLu/dbrtNQD+5fTzn3D3+5dtz1pMLIQQQgghhBA1ZlYAeDOA7wTwMICPmNkD7v7JUPSX3P11Yd9bALwRwN04+pnuo9N9n8ISKEGeEEIIIYQQXeNL/l2dFwH4jLs/5O4jAO8A8MqWrfsuAO939yenk4n3A3h5y30b0cRCCCGEEEKIrll+YnGbmT1If/eEI9wJ4PO0/vD0s8h/Y2Z/YGbvMrPnXuO+18R6SKGMsijHGAWWtnIW6qaM3PM4TjtM1nDnjptkqCZL3TiFzejeE9l/TpfOInOyV/SMj24vCAdZEj0ckjVkyEBbjqmRHNsRbhfbzcbMwL1Dsr29sdYyF4dpuYqtiKkZib0ugP5OXd+ZQeq9ecdOrfs+26/jHv7iIM2GPaZAhTuGdQxEGUT/nJW7mtlWa8KfofTNz0xS0f6IjsUWpfuTVNc9Otf8u0G5xT69jcUS2Ho02iOfdmZ0rkkW8Uy2bY4raDleY4yJUczBhJZHk7TT9zOPLI7H6FOG7t4wHV/VoH6kczhPDDdgW+Fov8zxSDkr7cTOmi9Nme50aURW1CGI50yvHlMcV3G+lzZ4YPXFqegCH3h6/tzjY3zTLgW/XChrr+BRaFOM47uM94MlOC2XMaRiOwTaCCE2kg5iLB5397uXrOP/BvB2dz80s/8WwP0AvmPpljWgNxZCCCGEEEJ0zeqlUI8AeC6t34U6SPuoCe5PuPvlX3HeCuCvt913EVY2sTCz55rZB8zsk2b2CTP7kennt5jZ+83s09P/33y1uoQQQgghhBAJHwHwQjN7gZkNAbwKwANcwMzuoNXvBfCp6fL7ALzMzG6efhd/2fSzpVilFGoC4H9w9981s/M4ijZ/P4AfAvBr7v4mM3sDgDcAeP1Va7ushIq2mUXDa+uYupbJ2WHmnGLZhnHVSpAGecaMHSZJJGbOmRMKl/OXj8pxJmuyZQ0nWVbN81Aum1XJtE5ZXlM13OOj+urFaEvLUijO7F310w6wvV3Ln778XGqG8JfOfuHK8o1FbWU5DPayD49uoeV6rnzb4GJSriD9yIXJ2WTb0+Na/nRxXNtcsvQJAC6M6nJ83Z8KWcP75+pz5izkAFANyG6U+0NUGnpD/4p96DTCHTEjd8rBFrNxfLFEj6/hTMZ7luGN63s5Dv0wZzfbNum58zgs+N6FgpnL0Rs3n3Najq1ouRFpuYNJ/c/MhTLto0+W564sP6d44srywNPxdUDrY7LHvRBkkl+i+r9QpjayD49uvbL8yGEtZXzs4FxSbn9c65pKkqvZjO8xLcd/l3YkhRJi42n/1mHxQ7hPzOx1OJoQFADuc/dPmNmPA3jQ3R8A8I/N7Htx9L38SRx9D4e7P2lm/wpHkxMA+HF3f3LZNq1sYuHujwJ4dLp80cw+haOgkFcCePG02P0APog2EwshhBBCCCHWhOPIY+Hu7wXw3vDZj9HyjwL40YZ97wNwX5ftOZbgbTN7PoBvAvDbAG6fTjoA4AsAbm/Y5x4A9wDA8KzUUkIIIYQQYo1YL5+TTlh58LaZnQPw7wH8d+7+DG9z98YXRe5+r7vf7e5397fPzisihBBCCCGEOCWs9I2FmQ1wNKn4t+7+H6Yff9HM7nD3R6cBJY+1qmw6/YivlVgD7P16ntQUe3G0seUUMqPLzoYHZCxgG2XfufOigjN2s6RtbhmyMEOiv6YYiyrEVJRVfYBJNt6C6o6Xmuqwis8xNorbFLbRtSkO64udjcVIGhhWqVP1wvV9Vr+OkXj+4PEry9uW2tJ+xbDuxp8d3XZl+c8On52U2yPfT46pAFLbywndlEvjVK994aCOv9g/rOs72EstOs/SfLx/KT3paovsRsletEhPK+l7vNyrqlAOJ0Ou3yTjq92YT+xlIzHGwucv5+ycK7KbLcu0Y4/LWs8f7Zyd2p/s1/LCx9gsJp5z7IAG4QAAIABJREFUb0QxFhynVKTtNeo3HHJkwW52f1THLDx+mMYzPLJdv41m69lbe/tJOR6Xh+Sj+0R1Jin3BMVs/MU4fdPNcRWf36+3fWH3hqTcxT2yhD6ke3LYPm7Ph+vh5C6EWC3HIYU6bazSFcoA/DyAT7n7/0KbHgDwmunyawC8Z1VtEEIIIYQQ4kRYvd3sqWOVP6t8K4C/C+APzexj08/+OYA3AXinmb0WwOcA/MAK2yCEEEIIIcTxssaTg2VYpSvUb6DZ1PCl11ZZbe3Ym4RNrApgqUKUu1THeHdZdlG1kyrMSDC6cCtkqUZLi92cKydn4i6DBIOlGz2yzZypr+E2xMzbyXqUiVGlxV7dIaob0/S3SXZozuQ9Ths1GpEdJtm8AsBFsq8cDOtjPaefSjXOW90Rv6L/2SvLfzxI7Wt/a/eFV5ajLOTCqD72LsmfntxP5R4s1Rix/OmZ5uFMycABhGtYK1ASaRkQ7GYXtGztms5tn/m9bU4K1ZZ4mbiRJBOqghQqkTgVzQOWJYpR4ZVYomZkUvw8YCkcAFhJG/n5tZO2l21p2W43SoYOaXw9NUrlf39OFrAD0lM9WRwk5Ub0QKjoof+lSSpjYjvbp8bpuPniYV32S/v12HtqN23T4cV6fNmluu39/SCFm8yXEwJXkeIKIcR1jISgQgghhBBCdIghmzLoukUTCyGEEEIIIbpGUqjTy+VX91GqkbyC5tfPUSIwaZYWdB6131YykjjJRLcnWqe7NJN5u2WvTaRgUZ7FkimSYMQmsRPUaJJql4re/Ha0NeCakULlMqfzcQ9q+cToptQVieUZBWUJ7h+kdR9cqveLDjF/euZZV5af1a/dks8GV6gz/bod53v1yXxZcSkpd9ewTmr5uf1bk22cUfvJ3VrGsbuX6uLG1F7bq481fDqVqrDEabDrYVtFy/X1GFwMKbVzErpFyFqGdVF/y2Is8ergHBNJZujLidSK5Ekz5lG8bSbLMzkw8XIYdxWtO2/LjKf4bGRZptHFqbbDiVUspyMpZJBCjfZrieLDF29Ktp0b1J20pIs4sLQfjumiXirr8bA7ScfGIWWpj25qTx3WY+rxS7Vl2t6FVApVPE3yp12buwykz5f+4QZ+exCngv+0+zUn3YQsn3o8TVU2eGzQUPL6RK5QQgghhBBCCLEAa/PGQgghhBBCiLVhA99YaGIhhBBCCCFE12hicToxrzXyxUGIsRhz5uVa2dUro21mvTyj36+WFFlfgw1no94u2Fyyzrm1dWFOR812kGXcRhaY1fzluD6apF1nSDEGiR1mqCOxw6TL7kWwa2wp0rPRpHkjh99QvEF/Ly3WI5vWL51LLWA/1rvryvJjB+evLH94+2JS7g7yc93q1fEXVTiRPz+85cpytN58hjJqX7xUbysvpprU4mKtI2fX22gpO7hYX4DhpbSP9/fqTtA/U7ex2A/Xk/oUx+nMOJkuYkUb91kk5qKLOjIk1rZhjCbOrgUvh3J97ogUvxCOZRkxbtGbn2GerZ0BoKRjVdRtqiBrriij9sxh6XnI8Rc2DtnWt+qTHuxTX3sm7fOTc5R5284n2z5FF/Hp83UMxDD4iu9N6riiivbZG6dxVWN69kzKNCZk77Bux94z9VjrPR2eZRSrNKjDqjDYi8/oerk4bH5+C9El5/48HV//229dm3v/cXPDJ9KHz5d/7KCh5HWIK8ZCCCGEEEIIIRZiLd5YCCGEEEIIsVZs4BuL9ZhYuF+xlS1GQbbEjoosacrIFo6VjDQjyQwd3h31WIJAlrreDyeSe+fUkKHYotqF1ssJVTiMUig+WHofxiQ7mFAdPkkbyPIBzlwb70/FFrs569lMBmFWIRUjspvdS+sbkHRjbGm23s+RtetfnK2taLe20ou4PajX+0V9kr3wHvRgXJ/Y3kFqh8m2t0nG3730Gg4uks0nvVUeXkiPtXWxvjj9vfR+FXu1XKs4rI/VG6c6Oe/Xx+b7tZD0aR2I58V9L9iyJv0ykVqGOkn+1BvU17fop9e6yGTbTptU11cE2dVkQHbRg3pbuZWeV0kKotz4YilUL5x/Sfaz/QOSQoV+WA3qcqNReqzHD268snzhfC3/G4bxxVnJ+bE5CbbXLL2Mzx4/qMsWu/W24ZNpue0nWdZVL7PcC0jv82A3vZeWjKPNstecIZGkdu1fvXn0xuE7xeR0P4t747A+KucXXITq9H9r30Qp1HpMLIQQQgghhFgnNnBioRgLIYQQQgghxNKs3RsLy7z6SlxLgqTBG9xtjh3O+EtttJhRm16ls/NV2Q+v/ltKUowvTXgT2WN5wriea6bSJwCkTojZhVmqMCG5D8bhvOi1bW/U3PZsJmOGHHL6wdFoslO3g6VQRZA0bD9e1zHjaLNTa0YmZ8lVZiutYzdx/qENM+mV68V4/gO6Vn2SO/WDiQbLnwqWoMTs2rRtsJdem1QaxytRkzbfnSzrehPlf21lU60z1je7U1nb1NuJdqn5/HNjtKm63LaCpFC9kDV7QBK6mMk+cYXKpAofD+o+Xw3nO0QBQEkyx2oYMrYfkOyIzz9IoVh6yM/U4cW07VxHb5Qeq7pAMsRzddv3w/hKsoizIi8+X1gaWabbEmc4yqK99XR4HjzFUihyT9sNz5ezLCEMMrm2Tn4bwPDp0Uk3QYgTQ1IoIYQQQgghxHI4NlIKpYmFEEIIIYQQXbOBEwvFWAghhBBCCCGWZi3eWJhTXECUF/t8nW/Vj9rbBk31NbbjShVdS2gz+m1LsoiHgINcO/g06ZyjLWuPdMpGuv9qnM47e1tkZRkuwGRct6sa1csWNNUF1d87nG+berTjfE310Y5kKdmjbOuHIXiE7DDZoi/aRvYPm/vDhGw6y226NoNgxcsjiU85E2NhoS+z7W+PYkIGu2m5PmWfT2JHDuN9pXKXUp0zx0iwhXM1bA5osdy46Tjj9XGSZBQP95W18rHP836czX4mdohtlTkrfbhmHFdRhIzanJV7SNsmZRijtK0kK9oYp8T9NT4rk1iaTKxAuUVjj8dXyFDNNrX9g2bb260n6DqFNnGMCMdfxXgxJm7j8cFtHAQr5q2nyDp6t/bKjDEmuX9TbNyhpaYQYi0xKMZCCCGEEEII0QWaWAghhBBCCCGWJfum/zplbSYWV6QbMSqk4Y3zsVrKZrNrt6wiKg5IgpBYGYbzSs4zs80oI20vZt6mzJhsNTkJspgJSTV6IeOvs7UjyZ+iLKTYJ/kT2T/OSKEyJJmC+81hQiz5YrvKwW5zBuX4EBheqpcruidRPsJyrZxMLsqfkm0skyKpEkufAKBPWX653EzWbL5O4bh2WN90tx1abm5fUsdJPixZNphpB59LbhwmUqi4MZFCxSzyPL7qz6M9cO+QxtRB8yP3oF9vY+vZmSZRX54ES+iSsk0bSRmjLIjbGzNvN1lYT84P534OpJbF/Ul6MLaYHVxKNsHpcnA7YoZmbhPvM3PDMv2X5Vo9qr8frJiTzMDJPQ52u7TM2cWPNip8UayG/n66/qzfzvmxnwY274v1prM2EwshhBBCCCHWAtnNCiGEEEIIIbpAwdtCCCGEEEKI5dHE4pTipGcuQxxBYgHbtQdsMznr2YVmqEHnjGq+jjxaxfI598K1cZIE9xJ70WC3SnEP5S7FEQxTnXBF3aUMMRYg3beRfS1bygJpXEV/lz8PbcrEXLAVJdujRovKJkvVInVeTew1q4y9ZkEa69b3OMa9JBbACNvmWydbuOdNFsvR4tKH9f2a3LSVbBs8QXEabc+Fj5tzno2XkOMgcvERi4zfsI83PMVn2kT7sZ1zvNbcp2IsAo/F/gHF8DyTHmpwth5HY6N7Uqb17VJMRDFoDsYxtpSdpPrqarf2ZR1wPFPQZfMYmLmXbOdMMUzx/Hk/jgOL8W1O8QwxNomvYRKzEqxdi116cPCmLkIZYr9r2Q+rrfpeVgPFVIjj4aY/2TvpJgiRRU9DIYQQQgghOsZ8ub9WxzB7uZn9sZl9xszeMGf7PzGzT5rZH5jZr5nZ82hbaWYfm/490MU5r2xiYWb3mdljZvZx+uwWM3u/mX16+v+bV3V8IYQQQgghTgxf8u8qmFkB4M0AvhvA1wF4tZl9XSj2ewDudve/CuBdAP4Nbdt392+c/n3vYieZskop1NsA/AyAX6DP3gDg19z9TdNZ1RsAvP6qNTm9Mo+vqRN5BtlGLiqLWmC/LoJzZtrLFqic4XeUyl3K7foW9sqQGZbkBD2WWUTZ0W69XiXSj3TeOSFXxpjJl20eWQrV38tIoUg+0g/ZehO5VlQC0bViCYIFKViP7SGpvhmphs2Xfhyt29xyiQVwrJP75CTjLxtIZVI5rVGDtCgcis+l3EpvWD/aY15r+xZl1dm7m8bvjOyKFvnWzdhUU8HwMwz3t8GlelxOttLH6vYTVMWErWdj9vb6npTD5vZyO3isAcCAxtvgYr08fCbKH9lvNsj16DkyurW2Ip6R9ZXzx1SUkxl4G9Jt1fxxHq2Tk31G7I8db4rNX0YqeURuzHOd/ByeNNtUR2KdQogN5BreOizBiwB8xt0fAgAzeweAVwL45JVmuH+Ayn8YwN9ZZYNW9vRz9w8BeDJ8/EoA90+X7wfwfas6vhBCCCGEECfG8m8sbjOzB+nvnnCEOwF8ntYfnn7WxGsB/DKtb0/r/bCZdfKd/LiDt29390eny18AcHtTwenFuwcAtrZvOoamCSGEEEIIcWp43N3v7qIiM/s7AO4G8O308fPc/REz+woAv25mf+juf7rMcU7MFcrd3az5JZG73wvgXgA4f8NdTWYvyWv843SFytJW0sHNzTizJNuiQiKTYTxxkOLM03sxDTM7S9UfF8HRaXyOpDU7yaZU4sCJsYMbDcuuONt2P8izCpJJZWVHxIwrFLnR9A7rBkbJVNJv4js8koNlM2qzsxDXXzXL0xCkazPOYJePG2RL7PbEkos4nBIJXbiGaRbtuYe9rmEZC0tkYuZ1JNnbmztAf78eOFvPhH5I2bF7JF0qLwUp1IBld+EArHij7lCM02LJmKKxNtxNz4uzz/d3U30SS4Fybk9G25LrGd2jWCY14+rX0PnC517QSZ/hMRnkTiT5m8ko3iAvzfb/zHmlbYzHkhRqVQwvpDds+IGzJ9SSHJOrFxHXPYZjkUI9AuC5tH7X9LO0LWZ/A8C/APDt7n5FlO7uj0z//5CZfRDANwFYamJx3E+/L5rZHQAw/f9jx3x8IYQQQgghVo/7cn9X5yMAXmhmLzCzIYBXAUjcnczsmwD8HIDvdffH6PObzWxrunwbgG8FxWYsynFPLB4A8Jrp8msAvOeYjy+EEEIIIcTKWbXdrLtPALwOwPsAfArAO939E2b242Z22eXpJwGcA/Dvgq3s1wJ40Mx+H8AHALzJ3ZeeWKxMCmVmbwfwYhwFnjwM4I0A3gTgnWb2WgCfA/ADqzq+EEIIIYQQ1zPu/l4A7w2f/Rgt/42G/X4TwF/puj0rm1i4+6sbNr30Wus60qktKVRbZfzFom3LOW8mtolsL5rZJ2qbSQNdsAVqGV5UcfUl29IGK0vShJdpIufEftaq+ctAais72Od4i7RgjzNlHzbHKZQ7nP02d3G47lT/mui+47GcylYtrWMzcS9puegPS+0fUBxFEWIs+g0a8/jzRnLPW2rbM3juUKckvKnRije3S8GWxfGeNOvoGbZYHT4T+ldV38s+jalyGGIsMrEePe6GDWMNAIoxx9XQ+JqxmKaYo2i/TM+R3iFlaM9YJxvHIsxksm5YjmWtuQ6+HlUSExViG7i6Kl7D+e2ficUYzB9fM2OGn3NhfKE4LQPi+uPWP1TmabEm1M5OG8WJBW8LIYQQQghxvRJ//NkENLEQQgghhBCia/TG4hRz+ebEcPPmBK3HRyYbeH4/3qd5m8fssi1hyQxLGnpRFkPrbFEbpUUsGaj6cdv8NvRiFl6WD9C22KYiY3Ppw/pgFS3nrHeTLNTFIG1TIgVrll0lUpAoY+Lz6mXa0ZD9F0gtKlmOMSv3aLClzfSTmazBC0g1jsE2b3VkZFGJjCun6YoWqGz1y/1knJYbXKxlR71xrWPqR8VQYisdxsOI6+fO1mztmpXk8fNlEPpNQ0btmX6Ysd9NC7YrlntuNlqJz2RUb7aHbRofcSwktrRcf2xeF5nihRDiOmN9JhZCCCGEEEKsCWv9g9yCaGIhhBBCCCFElzg28s3m9TuxOMks3Lljt+1kTcXim3+SLbAsaGY32tbbD5l2OTMuSTqKKNVJZDyNh8rDyoJec30VZ/9FKuOpWOJFEpGZ7NJNBOlDmUgkUgemJinEjMtSSyeonLNSm30iSR0xM3JWCrMBrjVLOkQB4Rrm5FQNsigg7St9GnszUh2qvzdKNZ69RGpFTk2hH5Y7tcwvJ3dj+dNMRm3u8iwt6sDpaKbPL/KPbtu+G8vRrU3dqMJuTZnCc22dccLagPElhLgqm/jG4rgT5AkhhBBCCCGuQ67fNxZCCCGEEEKcFBv4xkITCyGEEEIIITrkKLnzSbfi+NHE4rQQJbltHWszOvqKM8gWzTaMHKeQ6IuDpaodZPTGvE62qRwPEWFdetY2Mvhysu45sfnMXTOuvwrnX5KePaeNzggHW1tv8j65XfhYIXQk1YCzBW7mAsT2NZ3notbJGY7zwdo6A3jT9Y1J6TMZpRtjhCyT2Z6zoYeYIMvFcBQNYyrEPXBsUnKsGEcxad6WHreDWIEko/bMxuXrb2ImUzbZyLIldK5/Jv1EcRNCiGvAfSODtxVjIYQQQgghhFgavbEQQgghhBCiYySFOs1cfgudu0mn8U1116/BolKl1ywzcHJOZflEFeQNxtmrG7JwA0F2FO1WE6kRaz/QTMYOlCUYM1INlkJl7FZbHTceOtdeVq7kpFtt3TBzzc0da5Eu1bH9ZWvJ0TrQZEMamLGH5fe9PA6jfCiRF1K/tswYCh3ROdk21edhHPZYvpiR0yV152R8XTy/rsWmtcu6c2VZ1hSVa9Sm5J7E9/st3a2FEBuMJhZCCCGEEEKIZdnENxaKsRBCCCGEEEIsjd5YCCGEEEII0SWO9hLt64j1m1icFl3rqi3EmqTHGTvQGBMBslSsKN5ixnqSZcRJ/EJOGx3Wy3bXg20zs+/LuP6ysVRy/rn2Jtr5tnEf10LSjsWqWJprsLxtHSPBsSMLikVPezxGLq6C+0rsX0b9kjfNxjrVH1SD5mNxv+F4JiAd22wxXIzSco220jmr3EjTGMhdp0y8VLbutuOtrT1yWzLPnjRerN0+Qggxl82bV6zhxEIIIYQQQohTjmIshBBCCCGEEGIB1ueNxeVZX8bmM9UjZMq1PuYKppoLVMmv5mdkJSR96I1SzVCPMvty9mrOzhu3lWw9G1ULOcUIKyFysqhEdkX2taOYhXj+PkcbM9sWoW3f6MI2c1HJyJLEe8eJh3OZlxdpx6p/oWnbD9OdvHHVSCY4YynbZD0ayqZZ7tND85iq6IkbbWn5vHplWklvXB+7v1/Q8iQtd1g/AxJb2njNeH0mu3bDBY4y1OPMKLusZCrSVlLbZFEbt8XdTrn8TwhxTGxg5u31mVgIIYQQQgixJmyiFEoTCyGEEEIIIbrEoeDtU4t73qFoVXQhkZmpk/ejj8vmd/M+qKUPLFsC0ozaM9KKJBswfR7Pq8lwJUpkvGF53vrlOmJmYD5PWoxtZ1nXzLVhpyKWrlyDK1JCW3lSF9mrF5E/5ZRKi7apZZ9Nfm1JpIaLHTZ/sJbF2h47p1zjscFZreNOGVeopFgyVkJfZkcn7sozWsP5+wCA0XpxOH8MAanrWiLHif0kl5W7yeGs9XOuAxnqTJUtnapWmil8hRnEhRDiOmE9JhZCCCGEEEKsCYar2PZfp2hiIYQQQgghRNecltxrx8iJ2M2a2cvN7I/N7DNm9oaTaIMQQgghhBCrwqZS/kX/1pFjf2NhZgWANwP4TgAPA/iImT3g7p9ss3+0VE3qzmS/XVh/z3SRmr03Pz6A4yiOPphvnWvjELPAcQoh/qLcotgMOlY5DLEYM3aTLYhZiJN4EWpfaK9VNJdlTfk4ndZXw7rtMza6lJU4yVCciVNJznDBOIpFLCSvyRGi64cI1VcN06FebdXr5VZ9TxL9PtK+sa4PuXk02sjGcJ6kk2bqyFgs90Zk+5pkw27X1rhfuU3jeuY5xzu1q9viOdOY6tF5ZW2JO2DheKEVku3zmWdvn55Z18+oEUKIq3MSbyxeBOAz7v6Qu48AvAPAK0+gHUIIIYQQQnSPd/C3hpxEjMWdAD5P6w8D+OZYyMzuAXAPAGxt3Xg8LRNCCCGEEGJpXAnyThPufi+AewHAzL706x/8F58DcBuAx0+0YdcXup7douvZLbqe3aLr2T26pt2i69kt1/v1fN5JN+BqHEeCPDN7OYCfBlAAeKu7vyls3wLwCwD+OoAnAPwtd//sdNuPAngtgBLAP3b39y3bnpOYWDwC4Lm0ftf0s0bc/VkAYGYPuvvdK2zbRqHr2S26nt2i69ktup7do2vaLbqe3aLreQpY8RuLlnHLrwXwlLt/lZm9CsC/BvC3zOzrALwKwF8G8BwA/4+Z/SV3bw5mbsFJxFh8BMALzewFZjbE0Uk9cALtEEIIIYQQYl1pE7f8SgD3T5ffBeClZmbTz9/h7ofu/mcAPjOtbymO/Y2Fu0/M7HUA3oej1zb3ufsnjrsdQgghhBBCrASfdd1bgNvM7EFav3caKnCZNnHLV8pMv4NfAHDr9PMPh33vXLbBJxJj4e7vBfDeBXa99+pFxDWg69ktup7douvZLbqe3aNr2i26nt2i63nSLC+Fenzd5GwnkiBvUcIsTSyJrme36Hp2i65nt+h6do+uabfoenaLrucpYPV2s23ilq+UMbM+gBtxFMR9zTHPbViriYUQQgghhBAC+P/bu9dYuaoyDuPPP3JTkZuQWhMNiECCxBSCCFEIKmghBtAolxgpXoIkGCVeqcRLEBLUKMYPXgELCARUGpqIXASjHxSFYuUOBYQELG1QI0bEAH39MPvEoT1Thu45Z2Yzzy/ZmT1rzznzdmftOX1nrfVuhlu3vAJY0uy/D7ixqqppPz7J1kl2A/YA/tg2oIktNytJkiR1Vea4KtSgdctJzgRuqaoVwPnAxUnuB/5OL/mged0VwF3AM8CpbStCQUdGLJIsTnJvkvuTnD7ueLomyWuS/DrJXUnuTPLJpv0rSR5NsqrZjhx3rF2S5KEktzfn7pamback1ydZ3TzuOO44uyDJXn39cFWSJ5KcZh8dXpILkqxLckdf26z9MT3faT5Tb0uy3/gin0wDzuc3ktzTnLPlSXZo2ndN8p++fvr98UU+uQac04HXeJKlTR+9N8m7xhP15BpwPi/vO5cPJVnVtNtHx6Gq3TbUW9TVVbVnVe1eVWc3bV9qkgqq6qmqen9Vvb6qDqiqB/t+9uzm5/aqql+O4p888YlFX43eI4C9gROa2rsa3jPAp6tqb+BA4NS+c3huVS1qts1ZUD/t3tacu5nFVacDN1TVHsANzXM9j6q6d6Yf0ruJz5PA8uawfXQ4y4DFG7QN6o9H0Bv23gM4GfjePMXYJcvY+HxeD+xTVW8E7gOW9h17oK+fnjJPMXbNMjY+pzDLNb5Bjf3FwHeb/w/o/5axwfmsquP6Pkt/DlzZd9g+Op8KWN9y66CJTywYrkavNqGq1lTVrc3+v4C7GUFJMc2qv170hcAxY4ylq95B7w/gw+MOpEuq6rf0hrn7DeqPRwMXVc9NwA5JFs5PpN0w2/msquuq6pnm6U30FjtqSAP66CBzUmP/xWRT57O5T8GxwGXzGpSmXhcSi9lq9Pqf4s2UZFdgX+APTdPHm2H9C5y284IVcF2SlUlObtoWVNWaZv8xYMF4Quu043nuH0P76OYb1B/9XG3vw0D/1IHdkvwpyW+SHDyuoDpqtmvcPtrOwcDaqlrd12YfnUehSLXbuqgLiYVGJMm29IZGT6uqJ+hNf9gdWASsAb45xvC66K1VtR+9aSWnJjmk/2BTdaGbnwxjkl5Vi6OAnzZN9tERsT+OTpIz6E0xvaRpWgO8tqr2BT4FXJpku3HF1zFe43PjBJ77BY19dBzmYY3FpOlCYjEndXanTZIt6SUVl1TVlQBVtbaqnq2q9cCPcJj5BamqR5vHdfTWAxwArJ2ZUtI8rhtfhJ10BHBrVa0F++gIDOqPfq5upiQnAe8GPtAkazTTdf7W7K8EHgD2HFuQHbKJa9w+upnSu1fBe4HLZ9rso2NiYjGRhqnRq01o5lqeD9xdVd/qa++fU/0e4I4Nf1azS/LyJK+Y2QfeSe/89deLXgJcNZ4IO+s537LZR1sb1B9XACc21aEOBP7ZN2VKAyRZDHwOOKqqnuxr32VmYXGS19FbFP/g7L9F/TZxjc9Jjf0pcRhwT1U9MtNgH9V8mfj7WAyq0TvmsLrmLcAHgdtnSs8BX6BXYWsRvekRDwEfG094nbQAWN7L2dgCuLSqrklyM3BFko8AD9NbPKchNAna4Ty3H37dPjqcJJcBhwI7J3kE+DJwDrP3x6uBI+ktiH0S+NC8BzzhBpzPpcDWwPXNtX9TU13nEODMJE/Tq+VySlUNu0h5agw4p4fOdo3PVY39F5PZzmdVnc/G69TAPjr/ZqpCTZlUR4daJEmSpEm0/cteXQft+dFWv+PaP391ZV85+06Y+BELSZIkqXOm8Mv7LqyxkCRJkjThHLGQJEmSRqq7lZ3aMLGQJEmSRqkwsZAkSZI0AlNYFco1FpIkSZJac8RCkiRJGrFM4VQoRywkacIleVOS25Js09z1/c4k+4w7LknSJlS12zrIEQtJmnBVdXOSFcBZwEuBn1TVHWMOS5I0SAHru5kctGFiIUndcCZwM/AU8IkxxyJJ0kZMLCSpG14JbAtsCWwD/Hu84UiSBuvudKY2XGMhSd3wA+D5cGCJAAADEUlEQVSLwCXA18YciyTp+bjGQpI0aZKcCDxdVZcmeQnwuyRvr6obxx2bJGmAjiYHbZhYSNKEq6qLgIua/WeBN483IkmSNmZiIUmSJI2SVaEkSZIktVdQ68cdxLwzsZAkSZJGbQrXWFgVSpIkSVJrjlhIkiRJo+QaC0mSJEkjMYVToUwsJEmSpFGbwsTCNRaSJEmSWjOxkCRJkkaqeiMWbbYWkuyU5Pokq5vHHWd5zaIkv09yZ5LbkhzXd2xZkr8kWdVsi4Z5XxMLSZIkaZQKWL++3dbO6cANVbUHcEPzfENPAidW1RuAxcC3k+zQd/yzVbWo2VYN86YmFpIkSdKojXHEAjgauLDZvxA4ZuPw6r6qWt3s/xVYB+zS5k1NLCRJkqRRa59Y7Jzklr7t5Bfw7guqak2z/xiwYFMvTnIAsBXwQF/z2c0UqXOTbD3Mm1oVSpIkSZo8j1fV/oMOJvkV8KpZDp3R/6SqKsnAIZAkC4GLgSVVNTMHaym9hGQr4IfA54Ezny9gEwtJkiRppGrOb5BXVYcNOpZkbZKFVbWmSRzWDXjddsAvgDOq6qa+3z0z2vHfJD8GPjNMTE6FkiRJkkapoGp9q62lFcCSZn8JcNWGL0iyFbAcuKiqfrbBsYXNY+itz7hjmDc1sZAkSZJGbX2129o5Bzg8yWrgsOY5SfZPcl7zmmOBQ4CTZikre0mS24HbgZ2Bs4Z509QU3hVQkiRJmivbb7FLHbTdRoWYXpBr/3Heyk2tsZhErrGQJEmSRm0Kv7w3sZAkSZJGqWoUN7nrHBMLSZIkadSmcMTCxduSJEmSWnPEQpIkSRqxciqUJEmSpHZqKqdCmVhIkiRJo1TM+Z23J5FrLCRJkiS15oiFJEmSNGrlGgtJkiRJLRRQUzgVysRCkiRJGqWqqRyxcI2FJEmSpNYcsZAkSZJGzKlQkiRJktqbwqlQqSm8eYckSZI0V5JcA+zc8tc8XlWLRxHPfDGxkCRJktSai7clSZIktWZiIUmSJKk1EwtJkiRJrZlYSJIkSWrNxEKSJElSa/8DO42XAEee8O0AAAAASUVORK5CYII=\n" + }, + "metadata": {}, + "execution_count": 154 + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAFgCAYAAAAmWu9SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOy9e7As2VXe+a3Kqjrn3Ee/pabVLSQBMg9jA3aHGAZikJARAmsQM0NgydgWDjl6wmFhZjxhS9gONIOJCdlEDEMMGkSH6FETYyRk2Rr1OARCA5I1DAjUAgF6ABKNhLppqdWv2/eeV1Vlrvnj1L357VWV++atyjrn1K3v13GiMyt37tyZuXfe2pXf+pa5O4QQQgghhBBiGXon3QAhhBBCCCHE+qOJhRBCCCGEEGJpNLEQQgghhBBCLI0mFkIIIYQQQoil0cRCCCGEEEIIsTT9k26AEEIIIYQQ1xPf9ZKz/sST5VJ1fPQPDt/n7i/vqEnHgiYWQgghhBBCdMgTT5b4nfd9+VJ1FHd8+raOmnNsaGIhhBBCCCFEhziACtVJN+PYUYyFEEIIIYQQYmn0xkIIIYQQQohOcZS+eW8sNLEQQgghhBCiQ46kUL7SY5jZfQBeAeAxd//6Odv/KYAfnK72AXwtgGe5+5Nm9lkAFwGUACbufncXbZIUSgghhBBCiI6plvyvBW8D0Oga5e4/6e7f6O7fCOBHAfwnd3+Sirxkur2TSQWgiYUQQgghhBBrh7t/CMCTVy14xKsBvH2FzQGgiYUQQgghhBCd4nCUvtxfV5jZGRy92fj3SROBXzWzj5rZPV0dSzEWQgghhBBCdEwHMRa3mdmDtH6vu9+7QD3/JYD/L8igvs3dHzGzZwN4v5n90fQNyFJoYiGEEEIIIUSHOIBy+YnF4x3FP7wKQQbl7o9M//+Ymb0bwIsALD2xkBRKCCGEEEKIjqngS/11gZndCODbAbyHPjtrZucvLwN4GYCPd3E8vbEQQgghhBBizTCztwN4MY4kUw8DeCOAAQC4+1umxf4rAL/q7ru06+0A3m1mwNFc4Bfd/Ve6aJMmFkIIIYQQQnSIA50GYM89hvurW5R5G45safmzhwB8wyrapImFEEIIIYQQHbN5ebc1sRBCCCGEEKJTHN5F8PbaoeBtIYQQQgghxNLojYUQQgghhBBd4kC5eS8sNLEQQgghhBCiSxyKsRBCCCGEEEIsjaGEnXQjjh3FWAghhBBCCCGWRm8shBBCCCGE6BAHUCnGQgghhBBCCLEsmyiF0sRCCCGEEEKIDnFs5sRCMRZCCCGEEEKIpdEbCyGEEEIIITqm8s17Y6GJhRBCCCGEEB2yqVIoTSyEEEIIIYToEIeh3MCIg807YyGEEEIIIUTn6I2FEEIIIYQQHaMYCyGEEEIIIcRSKMZCCCGEEEII0QGG0jcv4mDzzlgIIYQQQgjROXpjIYQQQgghRIc4gGoDf7/XxEIIIYQQQoiOUYyFEEIIIYQQYincFWMhhBBCCCGEEAuhNxZCCCGEEEJ0TCUplBBCCCGEEGIZjvJYbJ4wSBMLIYQQQgghOkUxFkIIIYQQQgixEHpjIYQQQgghRIcoj4UQQgghhBCiE0pX8LYQQgghhBBiCRy2kcHbm3fGQgghhBBCrDlmdp+ZPWZmH2/Y/mIzu2BmH5v+/Rhte7mZ/bGZfcbM3tBVm/TGQgghhBBCiI6pVu8K9TYAPwPgFzJl/l93fwV/YGYFgDcD+E4ADwP4iJk94O6fXLZBmlgIIYQQQgjRIceRx8LdP2Rmz19g1xcB+Iy7PwQAZvYOAK8EsPTEQlIoIYQQQgghOsRhKH25PwC3mdmD9HfPAk35FjP7fTP7ZTP7y9PP7gTweSrz8PSzpdEbCyGEEEIIITqmA7vZx9397iX2/10Az3P3S2b2PQD+LwAvXLZROfTGQgghhBBCiOsMd3/G3S9Nl98LYGBmtwF4BMBzqehd08+WRm8shBBCCCGE6BB3oFx98HYWM/syAF90dzezF+HohcITAJ4G8EIzewGOJhSvAvC3uzimJhZCCCGEEEJ0iqHCahPkmdnbAbwYR7EYDwN4I4ABALj7WwB8P4B/aGYTAPsAXuXuDmBiZq8D8D4ABYD73P0TXbRJEwshhBBCCCE6xLH6Nxbu/uqrbP8ZHNnRztv2XgDv7bpNirEQQgghhBBCLI3eWAghhBBCCNExq85jcRrRxEIIIYQQQogOcRgqX22MxWlEEwshhBBCCCE6ZhPfWGzeGQshhBBCCCE6R28shBBCCCGE6BAHUJ1wHouTQBMLIYQQQgghOsVQrjiPxWlEEwshhBBCCCE6ZFPfWGzeGQshhBBCCCE6Zy3eWAyGZ3175+aTbsZixLdgvkAdi+wTD+1USayPthlvq7yxXLI8s07LFi4Ar/fqea0XYY7bq8tlT7/lW0bLnP4i9a07VjXco3jP1+16tB0rTecV+mtSXdtr0bZcrq252xDHXps6c8da9B4vst+Cz7JT3w8XPa8No9y+Pn/LLA6qk27CRnJw+DRG491T/XSQFOqUsr1zM/7af/7DAADvpTcp+SKc+we3a+IXZm4Gb1pwYsFf/Kxst5PFLyNcx6Re7o3LtNxhvW60zUbjtByvjyfJNudtFdUxGKSN2hrW+5zdubI8uXEnKVZtF/Vy0XzPnbblyhV0jh6/PNK/dV4c30Mg3q8murDBjuNmcLG+Xz6oL0DvIO0b6K3ZQ7HtM4D6AF8b76fnm/SpcC34viTXd8GJRa+cP16B5ueBZSb/bZ8hsc/HvtJI5hnYfLBFfyU55f1w5ocWWj7lTT9OLnzlmWR9nVUiRnOJG/907+QassH8zsd+9qSbkMXdNlIKtRYTC3OgODwaxTYJvwzQPzgz/8gSC305i/+YNfSP+EW17T8k8YtwUmeffs3v85egUI6+BMQvD8kX12G9UozS4/b4i8Q2d4mttBxf+yrcB17lL+qDIinW9KWl2krLTWhiEb/s8X3uHdYH7o/SNnFfSb5k5fpJbB9/AV2zLwh8/6tB2nF4olXyGCrTa+gt1ZKW+cI4Mz5a7JOjqb5ra1NDuV56vk79stwKX8Abxm8chz2eJOReNnDfi09mmscn9cdzpGtT9TP1FfxMaR5f4ElMPBT3Fd5nJZPR0/1KYPYHr7q9bfvrJjI+v77XZnjhdPdJcTooN3BisXlnLIQQQgghhOictXhjIYQQQgghxLrgAKoN1EJqYiGEEEIIIUSn2EZKodZjYuF+RbPKsQfAnJiLy7tcyyRxSQ1s1HI7muMemGLSvK0kzW41qJcnwVWjHDYHjRrF4bLOuxoFHfmNLbsBy6hj21u60RQUE9Eb18vFbhoozuvVMI2/YDjuw2JQOl37RA8fdPRcf67ftDYKaNmfjjNmI/ZDvh7JcoyJadlGX+BXmdw+bQPbZ1ysuP4koDoEZQ8phsnmjzUAmJypy01CjAWXTWM2QjvYdIu9DGLwNsVRFON0Y/+gXh9eqAv2d6MRQ73NYhxU0qjMdaM+UA3rZ8NMkDeV47iMmTillrE0re85Vx2fecdoiJAcN/RDxVUIIY7yWGzes2DzplJCCCGEEEKIzlnpxMLMbjKzd5nZH5nZp8zsW8zsFjN7v5l9evr/NU1QIYQQQgghxHxK9Jb6W0dWLYX6aQC/4u7fb2ZDAGcA/HMAv+bubzKzNwB4A4DXX7Wm6Vt9C/qBJqnRtb2Jbvn+vGz4PGdLm5EB5JLRFftUbkKyjSAzmOyQTCpINSxRQjRb1jbKB2byYtTLUQpVkJKJ3/xNttM2leRgW4zq5eGlVLZRHDbbw7LHf0EWs8VMuVoW0rtUH8y309waye2LifqSgs2b0CSTaiut6oLYD5O+l2sI7RKLcZ0dyL/a4m3HZMayOSfPSXJX8CnO5LGgbUGRVw55meSPmZwGiewsjkMaAmWaJiYZ2xPqv/399BHe36sfUiw7jHAbY86MtP82y8kac75kLHBzZTvJ13JCqoOY4JNzBUkWJcRm4rCNlEKtbGJhZjcC+C8A/BAAuPsIwMjMXgngxdNi9wP4INpMLIQQQgghhFgTqjV967AMqzzjFwD4EoD/w8x+z8zeamZnAdzu7o9Oy3wBwO0rbIMQQgghhBDiGFjlxKIP4K8B+Fl3/yYAuziSPV3B3R0NOiQzu8fMHjSzB0fj3RU2UwghhBBCiO5wB0q3pf7WkVXGWDwM4GF3/+3p+rtwNLH4opnd4e6PmtkdAB6bt7O73wvgXgC44fyd9eQjJ4Fnve6M9ypXHnfMncYCtLRXTMpFi8aG84zxBmxRia20LGu4exQfEvsqa8UTDXjGvpbjI2aORZrtGEdQkYbdCypXBo3ypKR90m192pbo6GN8BMffsKZ8lArYjSw1473zjMY8pUGXvmbMWKW2jXVIdloweKT1tb72Y89YpfI696EYR8C2zyH+otzibZljUbdku9n+QWgkdcsqxi8M6zZOqI0xTqOkvtw/oPijw/DcoNikXpkGj7ENdMJM/FFv7iKqxexmV01TTFNbi+k1/TdeCHGCbGKMxcreWLj7FwB83sy+evrRSwF8EsADAF4z/ew1AN6zqjYIIYQQQghx3BwFb/eW+ltHVu0K9cMA/u3UEeohAH8fR5OZd5rZawF8DsAPrLgNQgghhBBCiBWz0omFu38MwN1zNr10JcdLpBSZgh2/mcq+6cpagNLH0Q6z4VxmstryoaKMhewxS1qO2YVZnsF2sDlb2nKc1lGQrKM45EaERlKdh+ebZVde1A2eyfJtdDJ0nXpFkHBQncUu2YHGbO28XjRn+c7SIN1ZNDNwJ2TkRKfSArMDyUzj9Y39ixOxs4ot04YqPi15XLItbSiX7Jc5xaSOMPbYBpeli2w3DaTSwOKgrnCwHzJ579f7FYdBanipXi4OWDaYjhu+VolcMfatgsdelBpiZax6rGVlUixZzTyzhRDXN2XnWvvTz6rfWAghhBBCCLFRODYzxkITCyGEEEIIITrF1jZOYhnWZ2IxvTdRwtGo6DhJqQcfO/apar7EKRdG33bCG/tvk1NNFdQ+1XD+crkdZAu8X1ATFSOSVhzwcnN7WYI1OZtuYxer6JDDTli9Mcss0jrYJaraqQ/W2x8n5ayqT6ayMCQaZAw5iVPufrV1oFk5SbbtZkla68zbTftEmrI6X0v9OXL1N+5Dy6Ff85iKmbcbq2v570g1aN42I4XqNWybkeDQ2CPntvJSWmyrx9Kl9KR7Y5IhjsiBrQwXh47VI+lPtZWOoVTW2TJ7ewd9YeEfCnk8ZLRrG/hDpBBCXJX1mVgIIYQQQgixJlSKsRBCCCGEEEIsw+UEeavEzO4D8AoAj7n718/Z/oMAXo+j99sXAfxDd//96bbPTj8rAUzcfZ7Z0jWjiYUQQgghhBAdcwwxFm8D8DMAfqFh+58B+HZ3f8rMvhtHiae/mba/xN0f77JBazexyNlBnkoaktgCqUY3a/9JOn8r0/PndQsZb7lOjreItpkcc8Hbyq30WNUOZ7lOt01K0ltTjEX/UjqoOOaiomzCo6A3T+wwQ7bxcshWmXTckKEbTRmE4zWkGItesKItt/tUbr301tk2ndQwWnXsU6Ptb86mud3FsMxYzsbH8DZO+J15+nKsEwBUA4phSKxtY1+uD8BdOVpM8/XojYMVLcc0cYxR+AeS47ZsTEFRM9czk1Gdy57Us71tn1xFTJAQ4rrlKEHeav/Nc/cPmdnzM9t/k1Y/DOCulTYIK8y8LYQQQgghhFiY28zsQfq7Z4m6Xgvgl2ndAfyqmX10yXoT1u6NhRBCCCGEEKedDoK3H+8i9sHMXoKjicW30cff5u6PmNmzAbzfzP7I3T+07LE0seiAmYzXbXUmC8hCohyDbVmtRDMtrW0Twis8p8zWxQ2jZFuPLsJkVGs1DnfSLta/yDqOerE8n57YIdWx9VSwGKbsvZNtlnS0vJ4zFsC8nNG7cBviPe/gbSfXcazWs13ThS1tF83g6rMWwA0W0DMFr7I+pZe6GSeZspkohWJZU7kd6iBZom9RH+2H/lqy1oqlkM3+tf2DdNvwmfpBklybYLfrJJPyYUYyyP0hjr3cM+u4aNtfJX26bhh/x4VkfWc4bijZzJPPnKmX//xMpmTKzmP1uPmyD+9e83HF+nBaEuSZ2V8F8FYA3+3uT1z+3N0fmf7/MTN7N4AXAVh6YiEplBBCCCGEEB1TeW+pv2Uxsy8H8B8A/F13/xP6/KyZnb+8DOBlAD6+9AGhNxZCCCGEEEKsHWb2dgAvxlEsxsMA3ghgAADu/hYAPwbgVgD/ux29gb5sK3s7gHdPP+sD+EV3/5Uu2qSJRQcs/KYrk0E3kRPQ8kxiZHI4KoK7S0VvdxMHl3gsVoKQsqIXZApJ4t1w0ls7tTRqm5YPhqkOZFzU63ZQayt8Kz3Y/nPonMu0m/ZIheV0jtUwSLdYqsHuNpkLEF23WP5wnFKlzo91Gt9NLigtyTswzc+8ncsobk4dO5cZParkfP5yTq5YkvtZdHRi+VO5k1bi5+q08n0aK/1BOm6MLs7osD5YWaRjaDSpT3Swm550uV13lsEz1IaYhZ6vLz9fQoZu3i/nrNY1bcdNJ0oFyaTWmhc/59PXvM+f3XjrleU/sDvb7/jY2Ws+llhT/FhcoV59le3/AMA/mPP5QwC+YRVt0sRCCCGEEEKIDnEo87YQQgghhBCiA05D8PZxcxoFEkIIIYQQQog1Q28sFiQ7CW1po+mJBjxYqnJcRRLbkGp5OWtuzKDLWanduL5wLE5+S1PNmP23N6o3lqMwJyW3vbNbdRDEoEg14Jd6pAHfqzXgFjJ5D284vLJ8uJtqUvu79bGLQ94SYkxY983xFtF6k+Mqgt1sem2ojhNLXX0dseJMxnzvot6eh0BuHGZjLpKs0ZmC3KX4uMG+lTPR+3baDwfbdYzFubN1+vpokzno1fvtjuoBfKG3k5QbU4zF+EL6z8Bkh8YHn1YMPyrmx1jE36vSeDE0Yi0vZ1uO9YfCVWeUF0KsHafFbva40cRCCCGEEEKIjtHEQgghhBBCCLEUjtW7Qp1GNLG4Bhr7R3wNzkqAXCLnlpmBU8lF2Jaxm3WSF3lirxnKJXaQXHd6rOT8g5yqovWCNA07g0labjvN2H2ZskzlE1vDer/D82lDJmfqsltPcwNDpdEe83KxIkg12A90xlKUKz3dD4i1e36twqLTG1eaaXndoo1sMlZyEh+WPDXIDgHAByRXDPbLAxoPN+7UUqizg3Q8DXt1ua2iXj4Yp4/68VYtQ5ycSa8TZ7PnZ0NvlLYpkT81WM8C4XlzGjNvr4KGZ48QQlzvaGIhhBBCCCFEx8huVgghhBBCCLEcrhgLIYQQQgghxJLIFUocwXrgLiwEZzTFHPfQo+XmKtiu0YN2l2MseqNgNxs9NqeMQx2JZS3ZUFoZG0XxHNupOLqq6nMZlbWo/Oww1YD3i1qM3u/XdcQYi4JsM20naLv7A7SBr02+IMeYpMdKrn2wB10lDbfu9LKKeIll8YblWKxojj/KxRz1yOm1or7h4anK+1mmTWwJXQT7ZR43Q4oJOtMP46sh2CPa0u5y/FUYTpPterka1idWHKTxUok9bIMt81EltE/bMblunMb+L4QQJ4AmFkIIIYQQQnSM3lh0jJl9FsBFHHl/TNz9bjO7BcAvAXg+gM8C+AF3f2qV7RBCCCGEEOK4kN3s6niJuz9O628A8Gvu/iYze8N0/fXH0I5uWXGm1UZ5RrSbZeVWUEH0JvNlPEWqikiPRXVUg/RgvkXypKDVGZKtrNNAYlkUkEqckrqDfW1J0qr+IEihOHsxJwnOyBGy0iK+l7GOar7eI1ffBj5HjhW+vjP3gbcl2euDxImTQff5/ofqWGoYZI3sUsxtKjPPBnKDnZFWGckQYzfkccMuIyyLAoABrXO5OO6MpFXxEqY2smjESnoe0DX0VbigNF1TSZCEEKcY38AvBDEC4Dh4JYD7p8v3A/i+E2iDEEIIIYQQokNWPbFwAL9qZh81s3umn93u7o9Ol78A4PZ5O5rZPWb2oJk9OBrvrriZQgghhBBCdEcFW+pvHVm1FOrb3P0RM3s2gPeb2R/xRnd3i5qaetu9AO4FgBvO33kq3ncnLiiL3u/EgajZ7SlZzzjJZOUerJKgBs9IMNi1hqQa7BAFACAnmeFWqqdi1xm+pTHjL8OvCL1slkJZL550w/KKe0knTk05CZ1kHcvD17ciqc4kSPBIQ+ekcYquRTGbfXKohszbMbd8Ra5LLLsqDkOfH9Xr5Tj9zWdMkkJenoT03T2fLzVscog72ims83OuIbs2EK7pgLWWzYcSQohNwZXHonvc/ZHp/x8zs3cDeBGAL5rZHe7+qJndAeCxVbZBCCGEEEKI40YxFh1iZmfN7PzlZQAvA/BxAA8AeM202GsAvGdVbRBCCCGEEEIcD6t8Y3E7gHfb0evzPoBfdPdfMbOPAHinmb0WwOcA/MAK2yCEEEIIIcQxI7vZTnH3hwB8w5zPnwDw0lUdd2k6sJH1TB1GAmbWc1thseDc+nozWnHaZRJiLKiO1Ho2ZN6ezC9n49AmirnohbiHnUEdY8EacM7IDaR6w+QVYRh8o1FdRzwWGnaLsSOdx1xwDEuMiUniPk7mQRJl9Nls7m3jObqO+1hxjEnSH+gexfijdDxkxsYhj9fm4/YPeFymQQaTLerztKkapnUUBxRjcZjWMRrVj+qDrXr50ngrKXeGYjgmVcsX0hnL3mwfomdRvL5CCCE2UwqlzNtCCCGEEEJ0iGMzg7dPIo+FEEIIIYQQ4jpDbywWhWUbOXlH3MRTObarDDaXSTZg/rwI9pKU/ZaX48G8P1/6AQA9Ul30SP5UBN/M3mFdXxUyZfcpsy8v740GSbmSZFJJQvGYeXvC9pUxa/ACsouWWbNnYMlTkvH7BH+FaNv3NoD4Y1BqCZ2TQlEm6/78exzXe8F6lp1eWVo1fCbVTNnZ+da21SD0+W2SGh4Eu9mD+lG9t1WPqWeKbTTBv5SV4UKlMsSwY9suRXa+yGY5bzfgNvCHPSEFnbie8c10kdfEQgghhBBCiI5Z1yR3y6CJhRBCCCGEEB3iUPC2uBosp6FZqCO++qddglTFG5xqYmZgI31Sklw39tH5iXYBAD3a6Ifz2wCkUqiCsv8Wh0mxxLVmdJhKnNiBZquobaaK4OjEEqpEThXOoyJXKKSHQsGGOazoyGX8zWRNt4y0KLlWxyg7StyNcq9SrydZVFP7F32XzPcuSA1tTBJCkhdWoQ0sk7IgNeyN5rcrft6ndrAMsQr9erJDY28/bcf4TN252SHqAlIp1KCYb11VtnWICmQlf3Q5WFrmw3CssmMtwCnUFlyTvHKD2Xl8EtZPph2f/6rzyfoHhy88mYYIcR2iiYUQQgghhBCdojwWQgghhBBCiA44hS9YV44mFkIIIYQQQnSMYixEexriLYDgoJezckysZ0PcA+m5KyO7ytBHOVPwTJwG2WMWGZ1z1a+7Adtm9oLdLOu+D/bTrnMwqdeHveYUxTzInHXf4fxBVVRleixy7EziKqIe3BKL2czPBm1jLMTJEGMvMveyKTZlpm9U3M+ps83c7ubYhF7DmIrHKkYUf7Bff14O06AgjmmahCFkZPU8GdfjIV4aHocFXYAyjC/LBAUkMV2Z0AweU07WszZp/xPdshnrc7ENbf89bxsfcS3fD2JsnThieGF09ULHQP/SmWT9wsWdE2qJEMthZvcBeAWAx9z96+dsNwA/DeB7AOwB+CF3/93pttcA+JfToj/h7vd30SYlyBNCCCGEEKJD3I9+TF3mrwVvA/DyzPbvBvDC6d89AH4WAMzsFgBvBPDNAF4E4I1mdvMSp3sFTSyEEEIIIYTomMptqb+r4e4fAvBkpsgrAfyCH/FhADeZ2R0AvgvA+939SXd/CsD7kZ+gtEZSqFXAGbWD3WrrTKMks0hkC71mqU48FEuj4jamInvIql8vcxZuACgOqO69VMaxezi8sjygzNuTMp278rlUJdvNBqnGpDkbsg/YspNkYsFuNrH2zVle0v3yfphrs6RhkSisOHXP2AOLFTIjcaNN1Dd6hxkZXxHqoDpZuhhlMFx/cUjHCpm8i0Oqb5LW0aNt1bhufNlLO9iIJYn91NozaVODnBBIr03ebpblT7Q8Tq+hF5yiPHo9c2pzrjsca4Gx17UFrCxlhRDXSgfB27eZ2YO0fq+733sN+98J4PO0/vD0s6bPl0YTCyGEEEIIITqmg+Dtx9397i7aclxICiWEEEIIIcT1xyMAnkvrd00/a/p8aTSxEEIIIYQQokMcywVud2RV+wCAv2dH/GcALrj7owDeB+BlZnbzNGj7ZdPPlkZSqC6IIrpkNadRbq4j0fOyjWyIAWANNMcbAECfbTRJDz1jI3tQi6yrYV1f/zBowA/qbcVueqy9va0ry1v9dnaz1aSuwybNMRYWqmMNeDWg5RkNPK1UmeAGrm+r37htxhJ3jZixv6zWWDC+bNzLTH20WMY+X3c+jkUCQn+jPlocpLEN5TbZOY9pHE5C/BHHVZUxsIhsmmls+DAtN5rUY5ljLGJoQ6+oG1wO0nOu+vxMyfT5cn5chcVnVJH5/ep6+WlrE7NgCSGuyqqfDGb2dgAvxlEsxsM4cnoaAIC7vwXAe3FkNfsZHNnN/v3ptifN7F8B+Mi0qh9391wQeGs0sRBCCCGEEKJLfPUJ8tz91VfZ7gD+UcO2+wDc13Wbrpffi4QQQgghhBAniN5Y5Oji9XbOXjTJDN1um43TClkKNWuHSfvR51amdXDm4d6I7GaDW2X/oG5Ifz891sGlWpO0t1UvF0V6rJIlHnQslFEKxe2LFpX1Ikuhom0mGuxmZyxlybKzCtmQl7aHlb3saslk5W7Kwj0Db4s/LiUZusPNJGkUS83isQqysC236/4VE9Qn4zzK/wraSGMll0E7TWodJF5U3yRKoWhMlcPm5wtY4jSuB6wN0jHkZ0jymJPgZaShbVnkx0HZyAohVsYGPl80sRBCCCGEEKJjVi2FOo1oYiGEEEIIIUTHbKKvgyYWkUV6QZRj5OprKps7Lm+K7lG8HCROLOsp9mlbUHSkUiiSaoxTyRBnBryopcwAACAASURBVOYs3ADQu1Tvt79dO0Rtn0ktqKoy6pUutz3UR1m/+bgAGl8teiZiaEbG0bSt7b3MlctlED4t5DIqbzKxbyWOUUFCSK5pPmjOLp1kot6i8TUKY7mavwwEuQ47RIVfw7hY2fKXMu+l7WBJYUmuUFWQEBYNYyBKDXl8ZaVQmWeg5EpCCLEeaGIhhBBCCCFEhzgkhRJCCCGEEEIsi2MxR4k1Z+V2s2ZWmNnvmdl/nK6/wMx+28w+Y2a/ZGbDVbdBCCGEEEKI48R9ub915DjeWPwIgE8BuGG6/q8B/JS7v8PM3gLgtQB+9hjacWzk9MBOKuikXMY2M6076KEzGn6u38kC0g7TgAbWPReHdYVF0IAXpCkf7KbtHdP6iDINj/oh7iNmFJ7SC5m3Oa4iZgpnG1xejnEaDGf/jZa9STbzljpvj2J8vn+LxlU0ZJSOP3hYLpt7S2YycV8vLHQNfe7izKZwzWwy/0Z7jLGgZc7KXW6nv+v0xrRPtKIdz+9fZcjePaHM26OiXo6v45P9cs8rLhZjJ2jdKDDDB/PjqI7qixng55c7zpiKmb7RxbGv0+ElhBBXY6VvLMzsLgB/E8Bbp+sG4DsAvGta5H4A37fKNgghhBBCCHHs+JJ/a8iq31j8rwD+GYDz0/VbATzt7pd/tnsYwJ3zdjSzewDcAwBbWzeuuJlCCCGEEEJ0hSl4u0vM7BUAHnP3j5rZi691f3e/F8C9AHDD+TvXdN6Wj9tpnRk4A0ujZmQGtK0ieUIRpFCgrNS98XzrWQAoSI7R300bzNIozi5c9gdJOfQTbcmVxWivyfKn/n7YRpIRlmvFTMZJVvJ+c/bfxA4zkwEdmezKiTTqNMqMYpNOYRMbbXpz71VPiZ0vWzZXW+ljNel7JJ/qTdJO1N/ncZjeoP4B2b6ShKraS491SNaxVdV8kydsK30YJFksL6QmVv1gbTukY3P2+iCFapKMLUon/043SOaEEKJTNvDxsso3Ft8K4HvN7HsAbOMoxuKnAdxkZv3pW4u7ADyywjYIIYQQQgghjoGVxVi4+4+6+13u/nwArwLw6+7+gwA+AOD7p8VeA+A9q2qDEEIIIYQQx44fGWcs87eOnEQei9cDeIeZ/QSA3wPw81fdw+yKI0t0RTpWTvnr8xmJT0N25Wo7ve09cqph2UL/INUWOUkhoiyivEDLQ9Z4pbKIaptcsRqcboAghdqL7lS0zFKocdQnNaXobnedroU0M3KzG9HSdYf6uU+e6DPoOMdG1/KnRLo2o3FrhvtNxf06jJsGl6T+7iRdP1P/zjO4mN7MajB/7FUHad0luasdsiQrZNc22q+/lx6LpYZMzF5fDckJarv5nxIjqWXMSt5WDrpQ324rSQzlGp37hBDiWtnAZ8ixTCzc/YMAPjhdfgjAi47juEIIIYQQQpwM6/nWYRlWniBPCCGEEEIIcf1zElIoIYQQQgghrm8khRKL6nKXpW1mYA+v1VrHnHAG3WF4UXVAy6SHtlEqZi8OKP5ikLZjeJHsJkkDbp4eq6Ts3UZ68Jhdmy1mo93sYI+yg1N90b6Ts4hz/Em0A02I17MhdqLt280ZjXau3zTdy5b75DJyx6pjPM6p5lpiKrqO78iFX7D9cNaKmJaTDPBpLEb/Ur2+FcZXsh9lr5/spMfqU+xEucVjMi3HcUoxxqKg8cbxETPn1RCbZOXyQTDZmIpjtHNe2BL8lMfjiZTJuDlbvBBLsYHDXxMLIYQQQgghusRxwq4qJ4NiLIQQQgghhBBLs/5vLPg10ymZGLZ+jZ95RZ68gs/oMVgaFWVRLHdxUidUg3Q+aWQVWezXXpO9aJt5SJm3g/WkJ/IEzi6clpuc4fOvF4sghRpQZu/Bfiqt6NN6j+RaxTgtx1KurKUsK5zayjgy9Xlb6VJLiYTnpv9NUq3ATN/oOBvyibGgzKRR4hLvXU7+xknkeTxEmVnVUEfoQ2w/OzNG6Xb19+tt5XY4FCk62KI2luP6WBYFAP2Dur0FWc/aJOrpmvp5LEdbckOj7XOzi0zsTVnexUZw68dDH/348BiPrg63SWyiEnL9JxZCCCGEEEKcNjSxEEIIIYQQQiyNYiyEEEIIIYQQ4tq5vt5YZOItcpPGa7IRnHfYRa0RO7EebWdFy5p6i/ERZD/r497cfQCgN6KYi0zbe1RHcRhiLLbpWDStLcbp+bLOu7+bxnokFpjcxDLEn/TpXHL2qnSd2NYzRy4+IhenwfVnYzFIfz8TH8JNzNiBZmMu2sZYNMUELdqvmXip11h6nFjPhvPivlJx/4qhCFRu+Mw43YjBlaU+2T5XwZaWrZ6530S72ZL2s3DdixHFN5EFbi/06ySGi5sR7n9uW2MfLZr7UHbcLOAaar3QhnJ+uWtiE4XVa8S5z+9fvZAQHbDs98t15PqaWAghhBBCCHHSOBRjIYQQQgghhFgW28gYi/WbWKyDXKKtNSITz4vqcGTkLhn72kSeUbJFazj0FmXrPVtLLoq9SVIuqeMw3cb0yPa1OAgZuvvzM3THrNks1ekFiVNVsNyj/rzcTnUQXGcihYrWq7weJVMsQ6qa7wNnUeblGdlGPyOFom0l3ROPshC2SqV+4+GnkeKA7Es9Heo2qre1zt5+jBmPs3QgM2n9erptBuXEUjXKfebvMnNf+RkQjjWgschyqv5uGPO9+WMj2tdOzrBMMG1Gj2SJxQH7VKflEomeN8iigFTWlHt+Z/pXVv6U1NGuWFJ3lJDyY6RsPq/0mdJtm4QQYl1R8LYQQgghhBBd40v+tcDMXm5mf2xmnzGzN8zZ/lNm9rHp35+Y2dO0raRtDyxxpldYvzcWQgghhBBCnHZWHGNhZgWANwP4TgAPA/iImT3g7p+80gT3/57K/zCAb6Iq9t39G7ts0/pNLKIcoSkp9Uy5lu+j2zrfLFI3gjxpAQlKlLukWYPjwWgx40bD0iKWLTjJooBUWjPTRHadIleVXnSI6ZGjDbtRRfUItWMSZBzVkCRUTbINpJIslnHNOCLl7nlDn7KQldwOyMWnomMhwK5bwYHKSa7Uo/OvQpvKHZJJUcLYKCcbPLFb17Gd3ku0zTC+Sto24bQ67OTcjprKMbns9fGc2f2LXIyinIrHEcufyu20r3Ff6QVHtuKQZYjtblJWWsb9MufOxpUEZybjZ+CiGbqbTqWL/pV59gohNpjVPwteBOAz7v4QAJjZOwC8EsAnG8q/GsAbV9kgSaGEEEIIIYToEsfRLyHL/F2dOwF8ntYfnn42g5k9D8ALAPw6fbxtZg+a2YfN7PsWPNOE9XtjIYQQQgghxPXPbWb2IK3f6+73LljXqwC8y935nfDz3P0RM/sKAL9uZn/o7n+6cGuhiYUQQgghhBCd00GCvMfd/e7M9kcAPJfW75p+No9XAfhH/IG7PzL9/0Nm9kEcxV9s1sTiWuxWW9exANk3VBzO0IVF56J2oHxp+s3t4OuRZslNy7EtrY1CPENDJueZ69Tni0PWs8MQb0DtrULbixHb2fJyGgPCMRbIZZpmnXqwtrWKJvZsqTkIKX4phiGxnt0/TMtVdSyGD0PcQ6++Bj06FwttKrcoToMsP4uDIEx/9LF621b6ZtTGdf3cR2MS4sZ+vmgM0wmxkuyny55z7JI8BKrmDNU2mR9vAQAVZ5unTRw3AcQYi+aM2k1tiPWnVswxiCuTAT6JA6M4ihh/xDa6mazcPFZsHGOpuFzmecDHytxjfuZFO18hhABwHDEWHwHwQjN7AY4mFK8C8LdjITP7GgA3A/gt+uxmAHvufmhmtwH4VgD/ZtkGrd3EQgghhBBCiE3H3Sdm9joA7wNQALjP3T9hZj8O4EF3v2wh+yoA73BPfvn5WgA/Z2YVjn7aehO7SS3KVScWU2uq/9Pdn1r2YEIIIYQQQohucPf3Anhv+OzHwvr/OGe/3wTwV7puT5s3FrfjyBf3dwHcB+B9YcazetyvvF7PZj9NNoRsqrkWL3s61/AWPM3y3K4NbWUcSSZcpJKBxuMC6B2ShCYjLeuNOLt0kE+wTIiP2wuShoZ9YnZtliT190I72NqW2hElHclxWarQixovkk9Ea9dBPUTKbV5utsBlu93BxVSe1d+tpVC2P062GdnUOu/WT4/V368PwBnQo41ueeGZur6d5yfbes/QSpI1GiknZfV6nNm1W1fYbO2a3a0pY3tUO2XkPqnsqPlYydiIYyqpr/l50GS/PHM9WZ6VyV7fKLVEKiFi+VOUKvEzyhIb6WD7nDlnb7Cize6TyLNS+WNy7MFW2l7KbF9tbbYwYHRz7YldBsnrzhcPjrs5QhwrK5HhnnKu+pXY3f8lgBcC+HkAPwTg02b2P5vZV664bUIIIYQQQqwnq7ebPXW0+q19+obiC9O/CY4CQN5lZo1BHma2bWa/Y2a/b2afMLP/afr5C8zst6epx3/JzIZNdQghhBBCCLF2eAd/a0ibGIsfAfD3ADwO4K0A/qm7j82sB+DTAP5Zw66HAL7D3S+Z2QDAb5jZLwP4JwB+yt3fYWZvAfBaAD971ZZefvsd5Aj8ypmlBFV8bZ3Lyt2Wllm5WztBJVKCBZuUOVSTNCgrJ8tcG3ZCiq5IXKeTdCc6OqXyrExb+VJP2rnWVP2WmrQohSJnnXIrPa/Jdr1e7tB5BVOoJDs6NWOyk86b+/uUXXsU5BOcoZgXw2nx9e2xU1e4LgXdy4Nnp8caPEqyK5bdZPou1x9ld52wSvlTru62WZ2zB848D1h2k9zXqAui5di/+FB8n6OMh6VLaDeuZw9A1yP3L1tDn2+Up2KOmxq1q9gbXVnuBZkgxuySRheqysgf++HfAJZlFplnBV/DSeb8+dDx3yVq/6ZLoYQQm0Wbb2K3APiv3f273P3fufsYANy9AvCKpp38iEvT1cH0zwF8B4B3TT+/H0Anmf6EEEIIIYQ4NeiNxSzu/sbMtk/l9jWzAsBHAXwVgDfjKOnG0+5XQlMbU48LIYQQQgixrmxi8PZK39FO04Z/o5ndBODdAL6m7b5mdg+AewBga+vG1TRQCCGEEEKIVaCJxWpw96fN7AMAvgXATWbWn761aEw97u73ArgXAG644S6/nIk52pwmWtYhnc6wOQbgRGeQGavIhIZ4jqwGPKej5niO6HPJ+mBeHKdWjg62hgyxEwOOP6BYl5CRljNqJ7rsYEPI2bWjXs9GIcN0Exz3wXr20HaOx8nZyHJ7Z84/ybZeL5dn0nKjc/WxisNwziQrZwvMGGNRsrWt1/VVoU03D+r4jr1npePhPK9w7EQmq3HCKcm8vbCNdNdxFcn9bx5fVWLfGmxZc9mriTQ2o13jLT53ssfilN005uOh+Fyo/pg1m2OfioPUfjmJkeA6QryU0T9Vzgnrc/c4Wl1zjFguyzdnL0/GRhiItC3GZvVPyqZZCCFOmGvIwHBtmNmzpm8qYGY7AL4TwKcAfADA90+LvQbAe1bVBiGEEEIIIU4ExVh0yh0A7p/GWfQAvNPd/6OZfRLAO8zsJwD8Ho7yYwghhBBCCHFdYK4Yi05x9z8A8E1zPn8IwIuuqS6rM3YORtGWlLI3k/VseXaQlmuQ+8Q6siwg98hJGjqxwE0qbM4MnEgLwqF6SSZbsiHtR1kQyW6CjWyaQbf5WleD+oOSlnvR1tGaLWsHLOvi/WImY9qvIhkDtwEAJmdICrXVfI/ZYnbGRjeRSXHdQQpFGqRemW4r9uvl/j7ZcI6SYijJOZblT+VOWu7Ws/UHhzeF9pIVJ9+73iRkMqZrXZFcrZcZh63HScs+v4oHcyLryWSbzx2b62C5z4ycjO1XObt2uIStXx+zhDCMm2jN3Ngmtn2O9zI5Vua+0rVK5Fkzds714iQ8l7m9iRRsEq4GX7dc/2LL8UF8ftXrMQN00iZuB12aaImd9KEZ6+AN/DYhhJhlTZPcLcPKpFBCCCGEEEKIzUGZe4QQQgghhOiaDXx5qYmFEEIIIYQQHaMYi9OK1XrZKFdzthRkm8BwM1n37kGMz9aeK2eBWIqcVS5rm2csUFl/T5riImiqnbT+5Vatga6G0TaS9cu5GAMqF3pYUi5pOxrpBYdKtnYsynqjB001nzPHW7DlKwAc3sjnH61CaZHtMIOIkNf5nMfn03KTc9XccgDQ360P1t+rlwe74Vis7SfJ+vh80NufPVsf9yxSyGKXLXWxj1bEvha1/idG0/jK6fI5ZmfGeZXiCKJmvz8/Dihq+yc79bbJNsdipIdK4oxiO7gfUhhMjE3i2JzBbj02it1xUo7PswqxVD2O0+DljC1ruVUvH96SxlGMzjZfe26H5Vyk2cK5dlGeGUM5990Zy92G43JMU/+gbmB/v/mZ1xuFsTcODy0hxGZySv5pPE4UYyGEEEIIIYRYmvV4YyGEEEIIIcS6ILvZ04sb4NOWViGjNsufOLPqrN1fs2Qo+wp+lXhL6QNvC+fFZ+LBAnWyU1+PyZn6Oo2DjoclAmMqV27PbfXRsaJiiE+FZUEz8qz59Q2faT5WhM/FSraKDVIokp2MSY6yf1tabnRjvTy+IdVLVDusGeENoQ81PDxmpHtna4lEbzvteJMbqb17dF5PpRetN5qv95icTRvh587UdeyEbdu1XIWvW5S/sa0s23C2TPg8y2nJSJyxmGUSuV6wFE0lafV1O7g5vV8Ht7BMCnOXj9abpYE9UjIVB/Vyfy8tN7w4/x5ZsEpN72V6Xiw1TDJPh/Nnm+aDW+p/SnbvaB5f5VbohwOqP7FvRVqu73OX0Y+2x1TJJG1vjzLd9w5JdhgkTv1LtJxIEkN9JKHdejq0Q1KolfHMC3auXuiE2XmifrYPnhllSl47j3x7qmvdf85JfYFZjO3HMrrnljznN6YPwWjzfBo5Jf/kHSdrMbEQQgghhBBirdjAiYViLIQQQgghhBBLsyZvLOxKttUZpyKSRjm5lsSM11XOFYa3JVKjmA263Wu3XLbtheBMsKHumL2XKbfZCYmcaXbiNayXx+do/2HLrOEAigPK5EtvZnPuSVxH/yAtZyQ16oU3pyXJqwqSUrD0CQDGlPV6/1aSqjw7bfzo1rrBZ56VWjA9+4ZaF7FV1PKGwzIdOmOWZJEc43DcXK4XLuLOsNa78H5Pb59LytklcsXaJxnTTpBxnalTdM/ch2Fd/4T6ySDI6WzSss+3KrUCWo61meSn7ASVkfsgZqxuYHQDuSLdnO4zvoGWybmrDPfLt9h2LK3fxiTjobE2uBSdmmjclJwpPe2H/T0epOGZwm5P7KwWrg0/Xw5uqcvt357WV95xeGX5/A2p7diwX7djZ1D3/6KXXhseK33SbloYQ33aj8caAOyO6wfd03u1nGbvmVTzOXq61qgNn5zvLAcARm56w0vhQVxlHsxiKXbvTPv8wTfuNZQ8PiZPD5P15z+wumMd3pr2rbf9zZ9b3cE64C2PviRZ/60/+sprrmPwpcHVC51SNjHGQm8shBBCCCGEEEuzJm8shBBCCCGEWCP0xkIIIYQQQgghrp31eGNhtf3iTMbnQYN1WbBXTH1Z001JZutF2hfltJnpmmVCOBLYDZKz/4bst8jpw7k62hSzQbPum+1WfTucWEGNKoPemLT+/V22gw0NYZdPOpXhhWD5uFcfuzcOFrvJ9ag/n4Ss2YnVLR83dJnihtoO8Dk3pb63L7zhS1eWn7f9xJXlM73UQrCkA+yVdWzDXpVqb58en6Hl1Dbx0rje74ZhrUV/5OxNSbnPf+nmK8vjC/U+fB4AUJ5Lj80krpzUjBin0t9nLT5tyA2U2P9PSm7eMiaq7T4WrJ6dbFlLei5Nghvm6AaKq7ihjtMZhPvVKzh2IK2jpPE2oXiJw2HUHlMMz5gsVQ+CZfF4fowJkNpx8zMlZqVna2p+pkxuSq1W73z201eWv/qmx5Jtd2xfuLK8TZ66g4wH+JgGcO8afg68UNY35tGDOvDls2dvTco9Oqz9cQ8LilMq0ms4oHCsmWdvuV4WoOuM9TY7nuWsdWtnKzpEeSyEEEIIIYQQnaCJhRBCCCGEEGJpNLE4vVyWtVh4888WgDG77CpJMk3nFBctmzRjI0taCJY/zbxy58zAMRt2w7FjuXKbjn1jLUfY3hkn5YaDWuIQx8r+Xi0ZGG/VN8miBGPE51XXUg2DtIra3psEKRSvN9jXAkAxIrkHSUmCignj/XoYPLmX6lge36qznN5C2ocohXpW/+KV5bPDx68sl8Hn9cAp43W4EXtVfQ2/OK7lGF++81RSrqL9HkEtk7rlxtQqd3yulkzNyP+oT7G8sAz3oUmSd00Py9OSbZtozDydkzGOgj1srWpLLJujtW9Fts2DG2vr1XNnDpNybJ0ah/mErGMpGTRGO+n44qz047Mkn9pOK+zTPe+lwzy1mC24XHofe6R4SrryML1Ot+zUdqBfeeZLybav3n70yvL5Xi3/u6lILUS3SRrF/f/A03/CeHztepAhljSW+7egid1RXcfTLEE7SG1pe2TFXEaJ7kSZt4UQm8naTCyEEEIIIYRYBwyKsRBCCCGEEEJ0gSYWpxNnV6ggH6lI0tE/JCeOoCVgaVEvZkVNLFgW6AUzUo+MNmoBWUiS8TXjQNUrg2QokXtkmscKlwll6w1uG5wZelikricDWr/YqyUD434qR6j25rt4VaEnsnNTlGD0d2uZwfh8vWOUTDG8jbN6RzgTMADsTer2P7R725Xl4lx6bQY2X/owDhZUI2835M4UtUzmC4c3JtvuPFs76XAG4Ru30vTlezssoUvr5+zKFTUx3ocEPuWcUXXHisQZiV/OWW0BIyiu34Mdk03InWySOTHaRIq2ozq36z7V79cFi17a+JhtOqmes9RTHx0PQrZ1kl2xi9OMYxrJncybHfSMnim55wvLojBJOwc/swe9dJzcWtSZ7Z/Xrx3Zzofn9430TDn0+jl04Gmfv1DVcqqnq2ZXtIN+LXe6ZZiOr/Nbtczx4qA+7mQrvU6Ts5ShPFxfG6xvpmAhREcckyuUmb0cwE/jyBbwre7+prD9hwD8JIBHph/9jLu/dbrtNQD+5fTzn3D3+5dtz1pMLIQQQgghhBA1ZlYAeDOA7wTwMICPmNkD7v7JUPSX3P11Yd9bALwRwN04+pnuo9N9n8ISKEGeEEIIIYQQXeNL/l2dFwH4jLs/5O4jAO8A8MqWrfsuAO939yenk4n3A3h5y30b0cRCCCGEEEKIrll+YnGbmT1If/eEI9wJ4PO0/vD0s8h/Y2Z/YGbvMrPnXuO+18R6SKGMsijHGAWWtnIW6qaM3PM4TjtM1nDnjptkqCZL3TiFzejeE9l/TpfOInOyV/SMj24vCAdZEj0ckjVkyEBbjqmRHNsRbhfbzcbMwL1Dsr29sdYyF4dpuYqtiKkZib0ugP5OXd+ZQeq9ecdOrfs+26/jHv7iIM2GPaZAhTuGdQxEGUT/nJW7mtlWa8KfofTNz0xS0f6IjsUWpfuTVNc9Otf8u0G5xT69jcUS2Ho02iOfdmZ0rkkW8Uy2bY4raDleY4yJUczBhJZHk7TT9zOPLI7H6FOG7t4wHV/VoH6kczhPDDdgW+Fov8zxSDkr7cTOmi9Nme50aURW1CGI50yvHlMcV3G+lzZ4YPXFqegCH3h6/tzjY3zTLgW/XChrr+BRaFOM47uM94MlOC2XMaRiOwTaCCE2kg5iLB5397uXrOP/BvB2dz80s/8WwP0AvmPpljWgNxZCCCGEEEJ0zeqlUI8AeC6t34U6SPuoCe5PuPvlX3HeCuCvt913EVY2sTCz55rZB8zsk2b2CTP7kennt5jZ+83s09P/33y1uoQQQgghhBAJHwHwQjN7gZkNAbwKwANcwMzuoNXvBfCp6fL7ALzMzG6efhd/2fSzpVilFGoC4H9w9981s/M4ijZ/P4AfAvBr7v4mM3sDgDcAeP1Va7ushIq2mUXDa+uYupbJ2WHmnGLZhnHVSpAGecaMHSZJJGbOmRMKl/OXj8pxJmuyZQ0nWVbN81Aum1XJtE5ZXlM13OOj+urFaEvLUijO7F310w6wvV3Ln778XGqG8JfOfuHK8o1FbWU5DPayD49uoeV6rnzb4GJSriD9yIXJ2WTb0+Na/nRxXNtcsvQJAC6M6nJ83Z8KWcP75+pz5izkAFANyG6U+0NUGnpD/4p96DTCHTEjd8rBFrNxfLFEj6/hTMZ7luGN63s5Dv0wZzfbNum58zgs+N6FgpnL0Rs3n3Najq1ouRFpuYNJ/c/MhTLto0+W564sP6d44srywNPxdUDrY7LHvRBkkl+i+r9QpjayD49uvbL8yGEtZXzs4FxSbn9c65pKkqvZjO8xLcd/l3YkhRJi42n/1mHxQ7hPzOx1OJoQFADuc/dPmNmPA3jQ3R8A8I/N7Htx9L38SRx9D4e7P2lm/wpHkxMA+HF3f3LZNq1sYuHujwJ4dLp80cw+haOgkFcCePG02P0APog2EwshhBBCCCHWhOPIY+Hu7wXw3vDZj9HyjwL40YZ97wNwX5ftOZbgbTN7PoBvAvDbAG6fTjoA4AsAbm/Y5x4A9wDA8KzUUkIIIYQQYo1YL5+TTlh58LaZnQPw7wH8d+7+DG9z98YXRe5+r7vf7e5397fPzisihBBCCCGEOCWs9I2FmQ1wNKn4t+7+H6Yff9HM7nD3R6cBJY+1qmw6/YivlVgD7P16ntQUe3G0seUUMqPLzoYHZCxgG2XfufOigjN2s6RtbhmyMEOiv6YYiyrEVJRVfYBJNt6C6o6Xmuqwis8xNorbFLbRtSkO64udjcVIGhhWqVP1wvV9Vr+OkXj+4PEry9uW2tJ+xbDuxp8d3XZl+c8On52U2yPfT46pAFLbywndlEvjVK994aCOv9g/rOs72EstOs/SfLx/KT3paovsRsletEhPK+l7vNyrqlAOJ0Ou3yTjq92YT+xlIzHGwucv5+ycK7KbLcu0Y4/LWs8f7Zyd2p/s1/LCx9gsJp5z7IAG4QAAIABJREFUb0QxFhynVKTtNeo3HHJkwW52f1THLDx+mMYzPLJdv41m69lbe/tJOR6Xh+Sj+0R1Jin3BMVs/MU4fdPNcRWf36+3fWH3hqTcxT2yhD6ke3LYPm7Ph+vh5C6EWC3HIYU6bazSFcoA/DyAT7n7/0KbHgDwmunyawC8Z1VtEEIIIYQQ4kRYvd3sqWOVP6t8K4C/C+APzexj08/+OYA3AXinmb0WwOcA/MAK2yCEEEIIIcTxssaTg2VYpSvUb6DZ1PCl11ZZbe3Ym4RNrApgqUKUu1THeHdZdlG1kyrMSDC6cCtkqUZLi92cKydn4i6DBIOlGz2yzZypr+E2xMzbyXqUiVGlxV7dIaob0/S3SXZozuQ9Ths1GpEdJtm8AsBFsq8cDOtjPaefSjXOW90Rv6L/2SvLfzxI7Wt/a/eFV5ajLOTCqD72LsmfntxP5R4s1Rix/OmZ5uFMycABhGtYK1ASaRkQ7GYXtGztms5tn/m9bU4K1ZZ4mbiRJBOqghQqkTgVzQOWJYpR4ZVYomZkUvw8YCkcAFhJG/n5tZO2l21p2W43SoYOaXw9NUrlf39OFrAD0lM9WRwk5Ub0QKjoof+lSSpjYjvbp8bpuPniYV32S/v12HtqN23T4cV6fNmluu39/SCFm8yXEwJXkeIKIcR1jISgQgghhBBCdIghmzLoukUTCyGEEEIIIbpGUqjTy+VX91GqkbyC5tfPUSIwaZYWdB6131YykjjJRLcnWqe7NJN5u2WvTaRgUZ7FkimSYMQmsRPUaJJql4re/Ha0NeCakULlMqfzcQ9q+cToptQVieUZBWUJ7h+kdR9cqveLDjF/euZZV5af1a/dks8GV6gz/bod53v1yXxZcSkpd9ewTmr5uf1bk22cUfvJ3VrGsbuX6uLG1F7bq481fDqVqrDEabDrYVtFy/X1GFwMKbVzErpFyFqGdVF/y2Is8ergHBNJZujLidSK5Ekz5lG8bSbLMzkw8XIYdxWtO2/LjKf4bGRZptHFqbbDiVUspyMpZJBCjfZrieLDF29Ktp0b1J20pIs4sLQfjumiXirr8bA7ScfGIWWpj25qTx3WY+rxS7Vl2t6FVApVPE3yp12buwykz5f+4QZ+exCngv+0+zUn3YQsn3o8TVU2eGzQUPL6RK5QQgghhBBCCLEAa/PGQgghhBBCiLVhA99YaGIhhBBCCCFE12hicToxrzXyxUGIsRhz5uVa2dUro21mvTyj36+WFFlfgw1no94u2Fyyzrm1dWFOR812kGXcRhaY1fzluD6apF1nSDEGiR1mqCOxw6TL7kWwa2wp0rPRpHkjh99QvEF/Ly3WI5vWL51LLWA/1rvryvJjB+evLH94+2JS7g7yc93q1fEXVTiRPz+85cpytN58hjJqX7xUbysvpprU4mKtI2fX22gpO7hYX4DhpbSP9/fqTtA/U7ex2A/Xk/oUx+nMOJkuYkUb91kk5qKLOjIk1rZhjCbOrgUvh3J97ogUvxCOZRkxbtGbn2GerZ0BoKRjVdRtqiBrriij9sxh6XnI8Rc2DtnWt+qTHuxTX3sm7fOTc5R5284n2z5FF/Hp83UMxDD4iu9N6riiivbZG6dxVWN69kzKNCZk77Bux94z9VjrPR2eZRSrNKjDqjDYi8/oerk4bH5+C9El5/48HV//229dm3v/cXPDJ9KHz5d/7KCh5HWIK8ZCCCGEEEIIIRZiLd5YCCGEEEIIsVZs4BuL9ZhYuF+xlS1GQbbEjoosacrIFo6VjDQjyQwd3h31WIJAlrreDyeSe+fUkKHYotqF1ssJVTiMUig+WHofxiQ7mFAdPkkbyPIBzlwb70/FFrs569lMBmFWIRUjspvdS+sbkHRjbGm23s+RtetfnK2taLe20ou4PajX+0V9kr3wHvRgXJ/Y3kFqh8m2t0nG3730Gg4uks0nvVUeXkiPtXWxvjj9vfR+FXu1XKs4rI/VG6c6Oe/Xx+b7tZD0aR2I58V9L9iyJv0ykVqGOkn+1BvU17fop9e6yGTbTptU11cE2dVkQHbRg3pbuZWeV0kKotz4YilUL5x/Sfaz/QOSQoV+WA3qcqNReqzHD268snzhfC3/G4bxxVnJ+bE5CbbXLL2Mzx4/qMsWu/W24ZNpue0nWdZVL7PcC0jv82A3vZeWjKPNstecIZGkdu1fvXn0xuE7xeR0P4t747A+KucXXITq9H9r30Qp1HpMLIQQQgghhFgnNnBioRgLIYQQQgghxNKs3RsLy7z6SlxLgqTBG9xtjh3O+EtttJhRm16ls/NV2Q+v/ltKUowvTXgT2WN5wriea6bSJwCkTojZhVmqMCG5D8bhvOi1bW/U3PZsJmOGHHL6wdFoslO3g6VQRZA0bD9e1zHjaLNTa0YmZ8lVZiutYzdx/qENM+mV68V4/gO6Vn2SO/WDiQbLnwqWoMTs2rRtsJdem1QaxytRkzbfnSzrehPlf21lU60z1je7U1nb1NuJdqn5/HNjtKm63LaCpFC9kDV7QBK6mMk+cYXKpAofD+o+Xw3nO0QBQEkyx2oYMrYfkOyIzz9IoVh6yM/U4cW07VxHb5Qeq7pAMsRzddv3w/hKsoizIi8+X1gaWabbEmc4yqK99XR4HjzFUihyT9sNz5ezLCEMMrm2Tn4bwPDp0Uk3QYgTQ1IoIYQQQgghxHI4NlIKpYmFEEIIIYQQXbOBEwvFWAghhBBCCCGWZi3eWJhTXECUF/t8nW/Vj9rbBk31NbbjShVdS2gz+m1LsoiHgINcO/g06ZyjLWuPdMpGuv9qnM47e1tkZRkuwGRct6sa1csWNNUF1d87nG+berTjfE310Y5kKdmjbOuHIXiE7DDZoi/aRvYPm/vDhGw6y226NoNgxcsjiU85E2NhoS+z7W+PYkIGu2m5PmWfT2JHDuN9pXKXUp0zx0iwhXM1bA5osdy46Tjj9XGSZBQP95W18rHP836czX4mdohtlTkrfbhmHFdRhIzanJV7SNsmZRijtK0kK9oYp8T9NT4rk1iaTKxAuUVjj8dXyFDNNrX9g2bb260n6DqFNnGMCMdfxXgxJm7j8cFtHAQr5q2nyDp6t/bKjDEmuX9TbNyhpaYQYi0xKMZCCCGEEEII0QWaWAghhBBCCCGWJfum/zplbSYWV6QbMSqk4Y3zsVrKZrNrt6wiKg5IgpBYGYbzSs4zs80oI20vZt6mzJhsNTkJspgJSTV6IeOvs7UjyZ+iLKTYJ/kT2T/OSKEyJJmC+81hQiz5YrvKwW5zBuX4EBheqpcruidRPsJyrZxMLsqfkm0skyKpEkufAKBPWX653EzWbL5O4bh2WN90tx1abm5fUsdJPixZNphpB59LbhwmUqi4MZFCxSzyPL7qz6M9cO+QxtRB8yP3oF9vY+vZmSZRX54ES+iSsk0bSRmjLIjbGzNvN1lYT84P534OpJbF/Ul6MLaYHVxKNsHpcnA7YoZmbhPvM3PDMv2X5Vo9qr8frJiTzMDJPQ52u7TM2cWPNip8UayG/n66/qzfzvmxnwY274v1prM2EwshhBBCCCHWAtnNCiGEEEIIIbpAwdtCCCGEEEKI5dHE4pTipGcuQxxBYgHbtQdsMznr2YVmqEHnjGq+jjxaxfI598K1cZIE9xJ70WC3SnEP5S7FEQxTnXBF3aUMMRYg3beRfS1bygJpXEV/lz8PbcrEXLAVJdujRovKJkvVInVeTew1q4y9ZkEa69b3OMa9JBbACNvmWydbuOdNFsvR4tKH9f2a3LSVbBs8QXEabc+Fj5tzno2XkOMgcvERi4zfsI83PMVn2kT7sZ1zvNbcp2IsAo/F/gHF8DyTHmpwth5HY6N7Uqb17VJMRDFoDsYxtpSdpPrqarf2ZR1wPFPQZfMYmLmXbOdMMUzx/Hk/jgOL8W1O8QwxNomvYRKzEqxdi116cPCmLkIZYr9r2Q+rrfpeVgPFVIjj4aY/2TvpJgiRRU9DIYQQQgghOsZ8ub9WxzB7uZn9sZl9xszeMGf7PzGzT5rZH5jZr5nZ82hbaWYfm/490MU5r2xiYWb3mdljZvZx+uwWM3u/mX16+v+bV3V8IYQQQgghTgxf8u8qmFkB4M0AvhvA1wF4tZl9XSj2ewDudve/CuBdAP4Nbdt392+c/n3vYieZskop1NsA/AyAX6DP3gDg19z9TdNZ1RsAvP6qNTm9Mo+vqRN5BtlGLiqLWmC/LoJzZtrLFqic4XeUyl3K7foW9sqQGZbkBD2WWUTZ0W69XiXSj3TeOSFXxpjJl20eWQrV38tIoUg+0g/ZehO5VlQC0bViCYIFKViP7SGpvhmphs2Xfhyt29xyiQVwrJP75CTjLxtIZVI5rVGDtCgcis+l3EpvWD/aY15r+xZl1dm7m8bvjOyKFvnWzdhUU8HwMwz3t8GlelxOttLH6vYTVMWErWdj9vb6npTD5vZyO3isAcCAxtvgYr08fCbKH9lvNsj16DkyurW2Ip6R9ZXzx1SUkxl4G9Jt1fxxHq2Tk31G7I8db4rNX0YqeURuzHOd/ByeNNtUR2KdQogN5BreOizBiwB8xt0fAgAzeweAVwL45JVmuH+Ayn8YwN9ZZYNW9vRz9w8BeDJ8/EoA90+X7wfwfas6vhBCCCGEECfG8m8sbjOzB+nvnnCEOwF8ntYfnn7WxGsB/DKtb0/r/bCZdfKd/LiDt29390eny18AcHtTwenFuwcAtrZvOoamCSGEEEIIcWp43N3v7qIiM/s7AO4G8O308fPc/REz+woAv25mf+juf7rMcU7MFcrd3az5JZG73wvgXgA4f8NdTWYvyWv843SFytJW0sHNzTizJNuiQiKTYTxxkOLM03sxDTM7S9UfF8HRaXyOpDU7yaZU4sCJsYMbDcuuONt2P8izCpJJZWVHxIwrFLnR9A7rBkbJVNJv4js8koNlM2qzsxDXXzXL0xCkazPOYJePG2RL7PbEkos4nBIJXbiGaRbtuYe9rmEZC0tkYuZ1JNnbmztAf78eOFvPhH5I2bF7JF0qLwUp1IBld+EArHij7lCM02LJmKKxNtxNz4uzz/d3U30SS4Fybk9G25LrGd2jWCY14+rX0PnC517QSZ/hMRnkTiT5m8ko3iAvzfb/zHmlbYzHkhRqVQwvpDds+IGzJ9SSHJOrFxHXPYZjkUI9AuC5tH7X9LO0LWZ/A8C/APDt7n5FlO7uj0z//5CZfRDANwFYamJx3E+/L5rZHQAw/f9jx3x8IYQQQgghVo/7cn9X5yMAXmhmLzCzIYBXAUjcnczsmwD8HIDvdffH6PObzWxrunwbgG8FxWYsynFPLB4A8Jrp8msAvOeYjy+EEEIIIcTKWbXdrLtPALwOwPsAfArAO939E2b242Z22eXpJwGcA/Dvgq3s1wJ40Mx+H8AHALzJ3ZeeWKxMCmVmbwfwYhwFnjwM4I0A3gTgnWb2WgCfA/ADqzq+EEIIIYQQ1zPu/l4A7w2f/Rgt/42G/X4TwF/puj0rm1i4+6sbNr30Wus60qktKVRbZfzFom3LOW8mtolsL5rZJ2qbSQNdsAVqGV5UcfUl29IGK0vShJdpIufEftaq+ctAais72Od4i7RgjzNlHzbHKZQ7nP02d3G47lT/mui+47GcylYtrWMzcS9puegPS+0fUBxFEWIs+g0a8/jzRnLPW2rbM3juUKckvKnRije3S8GWxfGeNOvoGbZYHT4T+ldV38s+jalyGGIsMrEePe6GDWMNAIoxx9XQ+JqxmKaYo2i/TM+R3iFlaM9YJxvHIsxksm5YjmWtuQ6+HlUSExViG7i6Kl7D+e2ficUYzB9fM2OGn3NhfKE4LQPi+uPWP1TmabEm1M5OG8WJBW8LIYQQQghxvRJ//NkENLEQQgghhBCia/TG4hRz+ebEcPPmBK3HRyYbeH4/3qd5m8fssi1hyQxLGnpRFkPrbFEbpUUsGaj6cdv8NvRiFl6WD9C22KYiY3Ppw/pgFS3nrHeTLNTFIG1TIgVrll0lUpAoY+Lz6mXa0ZD9F0gtKlmOMSv3aLClzfSTmazBC0g1jsE2b3VkZFGJjCun6YoWqGz1y/1knJYbXKxlR71xrWPqR8VQYisdxsOI6+fO1mztmpXk8fNlEPpNQ0btmX6Ysd9NC7YrlntuNlqJz2RUb7aHbRofcSwktrRcf2xeF5nihRDiOmN9JhZCCCGEEEKsCWv9g9yCaGIhhBBCCCFElzg28s3m9TuxOMks3Lljt+1kTcXim3+SLbAsaGY32tbbD5l2OTMuSTqKKNVJZDyNh8rDyoJec30VZ/9FKuOpWOJFEpGZ7NJNBOlDmUgkUgemJinEjMtSSyeonLNSm30iSR0xM3JWCrMBrjVLOkQB4Rrm5FQNsigg7St9GnszUh2qvzdKNZ69RGpFTk2hH5Y7tcwvJ3dj+dNMRm3u8iwt6sDpaKbPL/KPbtu+G8vRrU3dqMJuTZnCc22dccLagPElhLgqm/jG4rgT5AkhhBBCCCGuQ67fNxZCCCGEEEKcFBv4xkITCyGEEEIIITrkKLnzSbfi+NHE4rQQJbltHWszOvqKM8gWzTaMHKeQ6IuDpaodZPTGvE62qRwPEWFdetY2Mvhysu45sfnMXTOuvwrnX5KePaeNzggHW1tv8j65XfhYIXQk1YCzBW7mAsT2NZ3notbJGY7zwdo6A3jT9Y1J6TMZpRtjhCyT2Z6zoYeYIMvFcBQNYyrEPXBsUnKsGEcxad6WHreDWIEko/bMxuXrb2ImUzbZyLIldK5/Jv1EcRNCiGvAfSODtxVjIYQQQgghhFgavbEQQgghhBCiYySFOs1cfgudu0mn8U1116/BolKl1ywzcHJOZflEFeQNxtmrG7JwA0F2FO1WE6kRaz/QTMYOlCUYM1INlkJl7FZbHTceOtdeVq7kpFtt3TBzzc0da5Eu1bH9ZWvJ0TrQZEMamLGH5fe9PA6jfCiRF1K/tswYCh3ROdk21edhHPZYvpiR0yV152R8XTy/rsWmtcu6c2VZ1hSVa9Sm5J7E9/st3a2FEBuMJhZCCCGEEEKIZdnENxaKsRBCCCGEEEIsjd5YCCGEEEII0SWO9hLt64j1m1icFl3rqi3EmqTHGTvQGBMBslSsKN5ixnqSZcRJ/EJOGx3Wy3bXg20zs+/LuP6ysVRy/rn2Jtr5tnEf10LSjsWqWJprsLxtHSPBsSMLikVPezxGLq6C+0rsX0b9kjfNxjrVH1SD5mNxv+F4JiAd22wxXIzSco220jmr3EjTGMhdp0y8VLbutuOtrT1yWzLPnjRerN0+Qggxl82bV6zhxEIIIYQQQohTjmIshBBCCCGEEGIB1ueNxeVZX8bmM9UjZMq1PuYKppoLVMmv5mdkJSR96I1SzVCPMvty9mrOzhu3lWw9G1ULOcUIKyFysqhEdkX2taOYhXj+PkcbM9sWoW3f6MI2c1HJyJLEe8eJh3OZlxdpx6p/oWnbD9OdvHHVSCY4YynbZD0ayqZZ7tND85iq6IkbbWn5vHplWklvXB+7v1/Q8iQtd1g/AxJb2njNeH0mu3bDBY4y1OPMKLusZCrSVlLbZFEbt8XdTrn8TwhxTGxg5u31mVgIIYQQQgixJmyiFEoTCyGEEEIIIbrEoeDtU4t73qFoVXQhkZmpk/ejj8vmd/M+qKUPLFsC0ozaM9KKJBswfR7Pq8lwJUpkvGF53vrlOmJmYD5PWoxtZ1nXzLVhpyKWrlyDK1JCW3lSF9mrF5E/5ZRKi7apZZ9Nfm1JpIaLHTZ/sJbF2h47p1zjscFZreNOGVeopFgyVkJfZkcn7sozWsP5+wCA0XpxOH8MAanrWiLHif0kl5W7yeGs9XOuAxnqTJUtnapWmil8hRnEhRDiOmE9JhZCCCGEEEKsCYar2PZfp2hiIYQQQgghRNecltxrx8iJ2M2a2cvN7I/N7DNm9oaTaIMQQgghhBCrwqZS/kX/1pFjf2NhZgWANwP4TgAPA/iImT3g7p9ss3+0VE3qzmS/XVh/z3SRmr03Pz6A4yiOPphvnWvjELPAcQoh/qLcotgMOlY5DLEYM3aTLYhZiJN4EWpfaK9VNJdlTfk4ndZXw7rtMza6lJU4yVCciVNJznDBOIpFLCSvyRGi64cI1VcN06FebdXr5VZ9TxL9PtK+sa4PuXk02sjGcJ6kk2bqyFgs90Zk+5pkw27X1rhfuU3jeuY5xzu1q9viOdOY6tF5ZW2JO2DheKEVku3zmWdvn55Z18+oEUKIq3MSbyxeBOAz7v6Qu48AvAPAK0+gHUIIIYQQQnSPd/C3hpxEjMWdAD5P6w8D+OZYyMzuAXAPAGxt3Xg8LRNCCCGEEGJpXAnyThPufi+AewHAzL706x/8F58DcBuAx0+0YdcXup7douvZLbqe3aLr2T26pt2i69kt1/v1fN5JN+BqHEeCPDN7OYCfBlAAeKu7vyls3wLwCwD+OoAnAPwtd//sdNuPAngtgBLAP3b39y3bnpOYWDwC4Lm0ftf0s0bc/VkAYGYPuvvdK2zbRqHr2S26nt2i69ktup7do2vaLbqe3aLreQpY8RuLlnHLrwXwlLt/lZm9CsC/BvC3zOzrALwKwF8G8BwA/4+Z/SV3bw5mbsFJxFh8BMALzewFZjbE0Uk9cALtEEIIIYQQYl1pE7f8SgD3T5ffBeClZmbTz9/h7ofu/mcAPjOtbymO/Y2Fu0/M7HUA3oej1zb3ufsnjrsdQgghhBBCrASfdd1bgNvM7EFav3caKnCZNnHLV8pMv4NfAHDr9PMPh33vXLbBJxJj4e7vBfDeBXa99+pFxDWg69ktup7douvZLbqe3aNr2i26nt2i63nSLC+Fenzd5GwnkiBvUcIsTSyJrme36Hp2i65nt+h6do+uabfoenaLrucpYPV2s23ilq+UMbM+gBtxFMR9zTHPbViriYUQQgghhBAC+P/bu9dYuaoyDuPPP3JTkZuQWhMNiECCxBSCCFEIKmghBtAolxgpXoIkGCVeqcRLEBLUKMYPXgELCARUGpqIXASjHxSFYuUOBYQELG1QI0bEAH39MPvEoT1Thu45Z2Yzzy/ZmT1rzznzdmftOX1nrfVuhlu3vAJY0uy/D7ixqqppPz7J1kl2A/YA/tg2oIktNytJkiR1Vea4KtSgdctJzgRuqaoVwPnAxUnuB/5OL/mged0VwF3AM8CpbStCQUdGLJIsTnJvkvuTnD7ueLomyWuS/DrJXUnuTPLJpv0rSR5NsqrZjhx3rF2S5KEktzfn7pamback1ydZ3TzuOO44uyDJXn39cFWSJ5KcZh8dXpILkqxLckdf26z9MT3faT5Tb0uy3/gin0wDzuc3ktzTnLPlSXZo2ndN8p++fvr98UU+uQac04HXeJKlTR+9N8m7xhP15BpwPi/vO5cPJVnVtNtHx6Gq3TbUW9TVVbVnVe1eVWc3bV9qkgqq6qmqen9Vvb6qDqiqB/t+9uzm5/aqql+O4p888YlFX43eI4C9gROa2rsa3jPAp6tqb+BA4NS+c3huVS1qts1ZUD/t3tacu5nFVacDN1TVHsANzXM9j6q6d6Yf0ruJz5PA8uawfXQ4y4DFG7QN6o9H0Bv23gM4GfjePMXYJcvY+HxeD+xTVW8E7gOW9h17oK+fnjJPMXbNMjY+pzDLNb5Bjf3FwHeb/w/o/5axwfmsquP6Pkt/DlzZd9g+Op8KWN9y66CJTywYrkavNqGq1lTVrc3+v4C7GUFJMc2qv170hcAxY4ylq95B7w/gw+MOpEuq6rf0hrn7DeqPRwMXVc9NwA5JFs5PpN0w2/msquuq6pnm6U30FjtqSAP66CBzUmP/xWRT57O5T8GxwGXzGpSmXhcSi9lq9Pqf4s2UZFdgX+APTdPHm2H9C5y284IVcF2SlUlObtoWVNWaZv8xYMF4Quu043nuH0P76OYb1B/9XG3vw0D/1IHdkvwpyW+SHDyuoDpqtmvcPtrOwcDaqlrd12YfnUehSLXbuqgLiYVGJMm29IZGT6uqJ+hNf9gdWASsAb45xvC66K1VtR+9aSWnJjmk/2BTdaGbnwxjkl5Vi6OAnzZN9tERsT+OTpIz6E0xvaRpWgO8tqr2BT4FXJpku3HF1zFe43PjBJ77BY19dBzmYY3FpOlCYjEndXanTZIt6SUVl1TVlQBVtbaqnq2q9cCPcJj5BamqR5vHdfTWAxwArJ2ZUtI8rhtfhJ10BHBrVa0F++gIDOqPfq5upiQnAe8GPtAkazTTdf7W7K8EHgD2HFuQHbKJa9w+upnSu1fBe4HLZ9rso2NiYjGRhqnRq01o5lqeD9xdVd/qa++fU/0e4I4Nf1azS/LyJK+Y2QfeSe/89deLXgJcNZ4IO+s537LZR1sb1B9XACc21aEOBP7ZN2VKAyRZDHwOOKqqnuxr32VmYXGS19FbFP/g7L9F/TZxjc9Jjf0pcRhwT1U9MtNgH9V8mfj7WAyq0TvmsLrmLcAHgdtnSs8BX6BXYWsRvekRDwEfG094nbQAWN7L2dgCuLSqrklyM3BFko8AD9NbPKchNAna4Ty3H37dPjqcJJcBhwI7J3kE+DJwDrP3x6uBI+ktiH0S+NC8BzzhBpzPpcDWwPXNtX9TU13nEODMJE/Tq+VySlUNu0h5agw4p4fOdo3PVY39F5PZzmdVnc/G69TAPjr/ZqpCTZlUR4daJEmSpEm0/cteXQft+dFWv+PaP391ZV85+06Y+BELSZIkqXOm8Mv7LqyxkCRJkjThHLGQJEmSRqq7lZ3aMLGQJEmSRqkwsZAkSZI0AlNYFco1FpIkSZJac8RCkiRJGrFM4VQoRywkacIleVOS25Js09z1/c4k+4w7LknSJlS12zrIEQtJmnBVdXOSFcBZwEuBn1TVHWMOS5I0SAHru5kctGFiIUndcCZwM/AU8IkxxyJJ0kZMLCSpG14JbAtsCWwD/Hu84UiSBuvudKY2XGMhSd3wA+D5cGCJAAADEUlEQVSLwCXA18YciyTp+bjGQpI0aZKcCDxdVZcmeQnwuyRvr6obxx2bJGmAjiYHbZhYSNKEq6qLgIua/WeBN483IkmSNmZiIUmSJI2SVaEkSZIktVdQ68cdxLwzsZAkSZJGbQrXWFgVSpIkSVJrjlhIkiRJo+QaC0mSJEkjMYVToUwsJEmSpFGbwsTCNRaSJEmSWjOxkCRJkkaqeiMWbbYWkuyU5Pokq5vHHWd5zaIkv09yZ5LbkhzXd2xZkr8kWdVsi4Z5XxMLSZIkaZQKWL++3dbO6cANVbUHcEPzfENPAidW1RuAxcC3k+zQd/yzVbWo2VYN86YmFpIkSdKojXHEAjgauLDZvxA4ZuPw6r6qWt3s/xVYB+zS5k1NLCRJkqRRa59Y7Jzklr7t5Bfw7guqak2z/xiwYFMvTnIAsBXwQF/z2c0UqXOTbD3Mm1oVSpIkSZo8j1fV/oMOJvkV8KpZDp3R/6SqKsnAIZAkC4GLgSVVNTMHaym9hGQr4IfA54Ezny9gEwtJkiRppGrOb5BXVYcNOpZkbZKFVbWmSRzWDXjddsAvgDOq6qa+3z0z2vHfJD8GPjNMTE6FkiRJkkapoGp9q62lFcCSZn8JcNWGL0iyFbAcuKiqfrbBsYXNY+itz7hjmDc1sZAkSZJGbX2129o5Bzg8yWrgsOY5SfZPcl7zmmOBQ4CTZikre0mS24HbgZ2Bs4Z509QU3hVQkiRJmivbb7FLHbTdRoWYXpBr/3Heyk2tsZhErrGQJEmSRm0Kv7w3sZAkSZJGqWoUN7nrHBMLSZIkadSmcMTCxduSJEmSWnPEQpIkSRqxciqUJEmSpHZqKqdCmVhIkiRJo1TM+Z23J5FrLCRJkiS15oiFJEmSNGrlGgtJkiRJLRRQUzgVysRCkiRJGqWqqRyxcI2FJEmSpNYcsZAkSZJGzKlQkiRJktqbwqlQqSm8eYckSZI0V5JcA+zc8tc8XlWLRxHPfDGxkCRJktSai7clSZIktWZiIUmSJKk1EwtJkiRJrZlYSJIkSWrNxEKSJElSa/8DO42XAEee8O0AAAAASUVORK5CYII=\n" + }, + "metadata": { + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "## Next steps\n", + "\n", + "* For this example, it's worth experimenting with various training parameters: run it longer, with varying learning rates, and different network sizes (or even different architectures)." + ], + "metadata": { + "id": "pC5xFvjv7wGz" + } + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 2 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython2", + "version": "2.7.6" + }, + "colab": { + "name": "Heat-SIP-forPBDL.ipynb", + "provenance": [], + "collapsed_sections": [] } - ], - "source": [ - "import matplotlib.pyplot as plt\n", - "\n", - "wct_times = {}\n", - "wct_times[mode] = wct_time\n", - "print(format(wct_times.keys()))\n", - "\n", - "wct_time = np.cumsum(time_list)\n", - "plt.plot(wct_time,loss_list)\n", - "plt.yscale('log')\n", - "plt.xlabel('Wall clock time')\n", - "plt.xlabel('Loss')\n", - "plt.title('Linear Chain - '+mode+'_'+str(optimizer.__name__))\n", - "plt.show()\n" - ] }, - { - "cell_type": "code", - "execution_count": 89, - "metadata": {}, - "outputs": [], - "source": [ - "path = './'\n", - "name1 = 'time'+mode\n", - "name2 = 'loss'+mode\n", - "np.savetxt(path+name1+'.txt',time_list)\n", - "np.savetxt(path+name2+'.txt',loss_list)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "interpreter": { - "hash": "60a46218545510a486eb79ffac0761a327b44b02462576ac242e6c15f2abc4b7" - }, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.5" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} + "nbformat": 4, + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/physgrad-comparison.ipynb b/physgrad-comparison.ipynb index 9f89300..a495de7 100644 --- a/physgrad-comparison.ipynb +++ b/physgrad-comparison.ipynb @@ -7,6 +7,7 @@ "# Simple Example comparing Different Optimizers\n", "\n", "The previous section has made many comments about the advantages and disadvantages of different optimization methods. Below we'll show with a practical example how much differences these properties actually make.\n", + "[[run in colab]](https://colab.research.google.com/github/tum-pbs/pbdl-book/blob/main/physgrad-comparison.ipynb)\n", "\n", "\n", "## Problem formulation\n", @@ -884,7 +885,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Nice! It works, just like the PG version above. Not much point plotting this, it's basically the same, but let's measure the difference. Below, we compute the MAE, which for this simple example turns out to be on the order of our floating point accuracy." + "This confirms that the approximate inversion works, in line with the regular PG version above. There's not much point plotting this, as it's basically the same, but let's measure the difference. Below, we compute the MAE, which for this simple example turns out to be on the order of our floating point accuracy." ] }, { diff --git a/physgrad-nn.md b/physgrad-nn.md index 506d130..a4d2e99 100644 --- a/physgrad-nn.md +++ b/physgrad-nn.md @@ -44,7 +44,7 @@ To update the weights $\theta$ of the NN $f$, we perform the following update st * Given a set of inputs $y^*$, evaluate the forward pass to compute the NN prediction $x = f(y^*; \theta)$ * Compute $y$ via a forward simulation ($y = \mathcal P(x)$) and invoke the (local) inverse simulator $P^{-1}(y; x)$ to obtain the step $\Delta x_{\text{PG}} = \mathcal P^{-1} (y + \eta \Delta y; x)$ with $\Delta y = y^* - y$ * Evaluate the network loss, e.g., $L = \frac 1 2 || x - \tilde x ||_2^2$ with $\tilde x = x+\Delta x_{\text{PG}}$, and perform a Newton step treating $\tilde x$ as a constant -* Use GD (or a GD-based optimizer like Adam) to propagate the change in $x$ to the network weights $\theta$ with a learning rate $\eta_{\text{NN}} +* Use GD (or a GD-based optimizer like Adam) to propagate the change in $x$ to the network weights $\theta$ with a learning rate $\eta_{\text{NN}}$ ``` @@ -74,11 +74,11 @@ The central reason for introducing a Newton step is the improved accuracy for th Unlike with regular Newton or the quasi-Newton methods from equation {eq}`quasi-newton-update`, we do not need the Hessian of the full system. Instead, the Hessian is only needed for $L(y)$. This makes Newton's method attractive again. -Even better, for many typical $L$ its computation can be completely forgone. +Even better, for many typical $L$ the analytical form of the Newton updates is known. E.g., consider the most common supervised objective function, $L(y) = \frac 1 2 | y - y^*|_2^2$ as already put to use above. $y$ denotes the predicted, and $y^*$ the target value. We then have $\frac{\partial L}{\partial y} = y - y^*$ and $\frac{\partial^2 L}{\partial y^2} = 1$. -Using equation {eq}`quasi-newton-update`, we get $\Delta y = \eta \cdot (y^* - y)$ which can be computed without evaluating the Hessian. +Using equation {eq}`quasi-newton-update`, we get $\Delta y = \eta \cdot (y^* - y)$ which can be computed right away, without evaluating any additional Hessian matrices. Once $\Delta y$ is determined, the gradient can be backpropagated to earlier time steps using the inverse simulator $\mathcal P^{-1}$. We've already used this combination of a Newton step for the loss and an inverse simulator for the PDE in {doc}`physgrad-comparison`. @@ -87,8 +87,8 @@ It is not to be confused with a traditional supervised loss in $x$ space. Due to the dependency of $\mathcal P^{-1}$ on the prediction $y$, it does not average multiple modes of solutions in $x$. To demonstrate this, consider the case that GD is being used as solver for the inverse simulation. Then the total loss is purely defined in $y$ space, reducing to a regular first-order optimization. -Hence, the proxy loss function simply connects the computational graphs of inverse physics and NN for backpropagation. +Hence, the proxy loss function simply connects the computational graphs of inverse physics and NN for backpropagation. ## Iterations and time dependence @@ -182,11 +182,12 @@ It provably converges when enough network updates $\Delta\theta$ are performed p While SIP training can find vastly more accurate solutions, there are some caveats to consider. % First, an approximately scale-invariant physics solver is required. While in low-dimensional $x$ spaces, Newton's method is a good candidate, high-dimensional spaces require some other form of inversion. -Some equations can locally be inverted analytically but for complex problems, domain-specific knowledge may be required. +Some equations can locally be inverted analytically but for complex problems, domain-specific knowledge may be required, +or we can employ to numerical methods (coming up). -Second, SIP uses traditional first-order optimizers to determine $\Delta\theta$. -As discussed, these solvers behave poorly in ill-conditioned settings which can also affect SIP performance when the network outputs lie on different scales. -Some recent works address this issue and have proposed network optimization based on inversion. +Second, SIP focuses on an accurate inversion of the physics part, but uses traditional first-order optimizers to determine $\Delta\theta$. +As discussed, these solvers behave poorly in ill-conditioned settings which can also affect SIP performance when the network outputs lie on very different scales. +Thus, we should keep inversion for the NN in mind as a goal. Third, while SIP training generally leads to more accurate solutions, measured in $x$ space, the same is not always true for the loss $L = \sum_i L_i$. SIP training weighs all examples equally, independent of their loss values. This can be useful, but it can cause problems in examples where regions with overly small or large curvatures $|\frac{\partial^2L}{\partial x^2}|$ distort the importance of samples.