{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "supported-manner",
   "metadata": {},
   "source": [
    "# A Teaser Example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lesbian-brave",
   "metadata": {},
   "source": [
    "Let's start with a very reduced example that highlights some of the key capabilities of physics-based learning approaches.\n",
    "\n",
    "Take a look at the following picture - the desired solution is shown in light gray. If we don't take care we'll learn an approximation like the red one shown on the left, which is completely off! With an improved learning setup, ideally by using a discretized numerical solver, we can at least accurately represent a part of the solutions (shown in green on the right).\n",
    "\n",
    "```{figure} resources/intro-teaser-side-by-side.png\n",
    "---\n",
    "height: 240px\n",
    "name: intro-teaser-side-by-side\n",
    "---\n",
    "Side by side - supervised versus differentiable physics training.\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deadly-paint",
   "metadata": {},
   "source": [
    "## Differentiable physics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "funky-tamil",
   "metadata": {},
   "source": [
    "One of the key concepts of the following chapters is what we'll call _differentiable physics_ (DP). This means that we use domain knowledge in the form of model equations, and then integrate discretized versions of these models into the training process. As implied by the name, having differentiable formulations is crucial for this process to support the training of neural networks.\n",
    "\n",
    "Let's illustrate the properties of deep learning via DP with the following example: We'd like to find an unknown function $f^*$ that generates solutions from a space $Y$, taking inputs from $X$, i.e. $f^*: X \\to Y$. In the following, we'll often denote _idealized_, and unknown functions with a $*$ superscript, in contrast to their discretized, realizable counterparts without this superscript. \n",
    "\n",
    "Let's additionally assume we have a generic differential equation $\\mathcal P^*: Y \\to Z$ (our _model_ equation), that encodes a property of the solutions, e.g. some real world behavior we'd like to match. Later on, $P^*$ will represent time evolutions, but it could also be a constraint for conservation of mass (then $\\mathcal P^*$ would measure divergence). But to keep things as simple as possible here, the model we'll look at in the following is a mapping back to the input space $X$, i.e. $\\mathcal P^*: Y \\to X$.\n",
    "\n",
    "Using a neural network $f$ to learn the unknown and ideal function $f^*$, we could turn to classic _supervised_ training to obtain $f$ by collecting data. This classical setup requires a dataset by sampling $x$ from $X$ and adding the corresponding solutions $y$ from $Y$. We could obtain these, e.g., by classical numerical techniques. Then we train the NN $f$ in the usual way using this dataset. \n",
    "\n",
    "In contrast to this supervised approach, employing differentiable physics takes advantage of the fact that we can directly use a discretized version of the physical model $\\mathcal P$ and employ it to guide the training of $f$. I.e., we want $f$ to _interact_ with our _simulator_ $\\mathcal P$. This can vastly improve the learning, as we'll illustrate below with a very simple example (more complex ones will follow later on).\n",
    "\n",
    "Note that it order for the DP approach to work, $\\mathcal P$ has to differentiable, as implied by the name. These differentials, in the form of a gradient, are what's driving the learning process.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "recreational-table",
   "metadata": {},
   "source": [
    "## Finding the inverse function of a parabola"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "latest-amino",
   "metadata": {},
   "source": [
    "To illustrate these two approaches, we consider the following simplified setting: Given the function $\\mathcal P: y\\to y^2$ for $y$ in the intverval $[0,1]$, find the unknown function $f$ such that $\\mathcal P(f(x)) = x$ for all $x$ in $[0,1]$. Note: to make things a bit more interesting, we're using $y^2$ here instead of the more common $x^2$ parabola, and the _discretization_ is simply given by representing the $x$ and $y$ via floating point numbers in the computer for this simple case.\n",
    "\n",
    "We know possible solutions for $f$ are the positive or negative square root function (for completeness: piecewise combinations would also be possible).\n",
    "We can try to solve this by using a neural network to approximate the inverse mapping $f$.\n",
    "Let's start with the supervised approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "accompanied-anaheim",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "numerous-emphasis",
   "metadata": {},
   "source": [
    "For supervised training, we can employ use our solver $\\mathcal P$ for the problem to pre-compute the solutions we need for training: We randomly choose between the positive and the negative square root. This resembles the  general case, where we would use optimization techniques to compute this the solutions. These methods are not expected to favor one particular mode in multimodal solutions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "realistic-event",
   "metadata": {},
   "outputs": [],
   "source": [
    "# X-Data\n",
    "N = 200\n",
    "X = np.random.random(N)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "capable-month",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generation Y-Data\n",
    "sign = (- np.ones((N,)))**np.random.randint(2,size=N)\n",
    "Y = np.sqrt(X) * sign"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stone-science",
   "metadata": {},
   "source": [
    "Now we can define a network, loss, and training configuration. We'll use a simple `keras` model with three hidden layers, ReLU activations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "weighted-costa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Neural network\n",
    "act = tf.keras.layers.ReLU()\n",
    "model_sv = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Dense(8, activation=act),\n",
    "  tf.keras.layers.Dense(8, activation=act),\n",
    "  tf.keras.layers.Dense(1,activation='linear')])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "confirmed-cargo",
   "metadata": {},
   "source": [
    "And we can start training via a simple mean squared error loss, using keras `fit` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "adolescent-yellow",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "40/40 [==============================] - 0s 842us/step - loss: 0.5951\n",
      "Epoch 2/5\n",
      "40/40 [==============================] - 0s 1ms/step - loss: 0.5594\n",
      "Epoch 3/5\n",
      "40/40 [==============================] - 0s 1ms/step - loss: 0.5462\n",
      "Epoch 4/5\n",
      "40/40 [==============================] - 0s 1ms/step - loss: 0.5404\n",
      "Epoch 5/5\n",
      "40/40 [==============================] - 0s 965us/step - loss: 0.5379\n"
     ]
    }
   ],
   "source": [
    "# Loss function\n",
    "loss_sv = tf.keras.losses.MeanSquaredError()\n",
    "model_sv.compile(optimizer='adam', loss=loss_sv)\n",
    "\n",
    "# Training\n",
    "results_sv = model_sv.fit(X, Y, epochs=5, batch_size= 5,verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "governmental-mixture",
   "metadata": {},
   "source": [
    "As both model and data set are very small, the training converges very quickly, but if we inspect the predictions of the network, we can see that it is nowhere near the solution we were hoping to find: it averages between the data points on both sides of the x-axis and therefore, fails to find satisfying solutions to our above problem.\n",
    "\n",
    "The following plots nicely highlights this: it shows the data in light gray, and the supervised solution in red. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "sought-basement",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 19608 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Results\n",
    "plt.plot(X,Y,'.',label='Data points', color=\"lightgray\")\n",
    "plt.plot(X,model_sv.predict(X),'.',label='Supervised', color=\"red\")\n",
    "plt.xlabel('y')\n",
    "plt.ylabel('x')\n",
    "plt.title('Standard approach')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "reduced-airplane",
   "metadata": {},
   "source": [
    "😱 This is obviously completely wrong! The red solution is nowhere near one of the two modes of our solution shown in gray.\n",
    "\n",
    "Note that the red line is often not perfectly at zero, which is where the two modes of the solution should average out in the continuous setting. This is caused by the relatively coarse sampling with only 200 points in this example.\n",
    "<br>\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dated-requirement",
   "metadata": {},
   "source": [
    "## A differentiable physics approach"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acoustic-review",
   "metadata": {},
   "source": [
    "Now let's apply a differentiable physics approach to find $f$: we'll directly include our discretized model $\\mathcal P$ in the training. \n",
    "\n",
    "There is no real data generation step; we only need to sample from the $[0,1]$ interval. We'll simply keep the same $x$ locations used in the previous case, and a new instance of a model with the same architecture as before `model_dp`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "extensive-forward",
   "metadata": {},
   "outputs": [],
   "source": [
    "# X-Data\n",
    "# X = X , we can directly re-use the X from above, nothing has changed...\n",
    "# Y is evaluated on the fly\n",
    "\n",
    "# Model\n",
    "model_dp = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Dense(8, activation=act),\n",
    "  tf.keras.layers.Dense(8, activation=act),\n",
    "  tf.keras.layers.Dense(1, activation='linear')])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "conscious-budapest",
   "metadata": {},
   "source": [
    "The loss function is the crucial point for training: we directly incorporate the function f into the loss. In this simple case, the `loss_dp` function simply computes the square of the prediction `y_pred`. \n",
    "\n",
    "Later on, a lot more could happen here: we could evaluate finite difference stencils on the predicted solution, or compute a whole implicit time-integration step of a solver. Here we have a simple _mean-squared error_ term of the form $|y_{\\text{pred}}^2 - y_{\\text{true}}|^2$, which we are minimizing during training. It's not necessary to make it so simple: the more knowledge and numerical methods we can incorporate, the better we can guide the training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "western-leader",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Loss\n",
    "mse = tf.keras.losses.MeanSquaredError()\n",
    "def loss_dp(y_true, y_pred):\n",
    "    return mse(y_true,y_pred**2)\n",
    "\n",
    "model_dp.compile(optimizer='adam', loss=loss_dp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "artistic-table",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "40/40 [==============================] - 0s 1ms/step - loss: 0.2260\n",
      "Epoch 2/5\n",
      "40/40 [==============================] - 0s 1ms/step - loss: 0.0139\n",
      "Epoch 3/5\n",
      "40/40 [==============================] - 0s 1ms/step - loss: 0.0015\n",
      "Epoch 4/5\n",
      "40/40 [==============================] - 0s 1ms/step - loss: 0.0014\n",
      "Epoch 5/5\n",
      "40/40 [==============================] - 0s 1ms/step - loss: 0.0014\n"
     ]
    }
   ],
   "source": [
    "#Training\n",
    "results_dp = model_dp.fit(X, X, epochs=5, batch_size= 5,verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "spatial-agency",
   "metadata": {},
   "source": [
    "Now the network actually has learned a good inverse of the parabola function! The following plot shows the solution in green."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "indonesian-abraham",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 20672 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Results\n",
    "plt.plot(X,Y,'.',label='Datapoints', color=\"lightgray\")\n",
    "#plt.plot(X,model_sv.predict(X),'.',label='Supervised', color=\"red\") # optional for comparison\n",
    "plt.plot(X,model_dp.predict(X),'.',label='Diff. Phys.', color=\"green\") \n",
    "plt.xlabel('x')\n",
    "plt.ylabel('y')\n",
    "plt.title('Differentiable physics approach')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prostate-radio",
   "metadata": {},
   "source": [
    "This looks much better 😎, at least in the range of 0.1 to 1. \n",
    "\n",
    "What has happened here?\n",
    "\n",
    "- We've prevented an undesired averaging of multiple modes in the solution by evaluating our discrete model w.r.t. current prediction of the network, rather than using a pre-computed solution. This let's us find the best single mode near the network prediction, and prevents an averaging of the modes that exist in the solution manifold.\n",
    "\n",
    "- We're still only getting one side of the curve! This is to be expected, because we're representing the solutions with a deterministic function. Hence we can only represent a single mode. Interestingly, whether it's the top or bottom mode is determined by the random initialization of the weights in $f$ - run the example a couple of time to see this effect in action. To capture multiple modes we'd need to extend the model to capture the full distribution of the outputs and parametrize it with additional dimensions.\n",
    "\n",
    "- The region with $x$ near zero is typically still off in this example. The model primarily learns a linear approximation of one half of the parabola here. This is primarily caused by the weak neural network: it is very small and shallow. (Give it a try - it's very easy to fix this in the `model_dp` definition.)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "necessary-filename",
   "metadata": {},
   "source": [
    "## Discussion\n",
    "\n",
    "It's a very simple example, but it very clearly shows a failure case for supervised learning. While it might seem very artificial on first sight, many practical PDEs exhibit a variety of these modes, and it's often not clear where (and how many) exist in the solution manifold we're interested in. Using supervised learning is very dangerous in such cases - we might simply and unknowingly _blur_ out these different modes.\n",
    "\n",
    "Good and obvious example are bifurcations in fluid flows - the smoke rising above a candle will start out straight, and then, due to tiny perturbations in its motion, start oscillating in a random direction. The images below illustrate this case via _numerical perturbations_: the perfectly symmetric setup will start turning left or right, depending on how the approximation errors build up. Similarly, we'll have different modes in all our numerical solutions, and typically it's important to recover them, rather than averaging them out. Hence, we'll show how to leverage training via _differentiable physics_ in the following chapters for more practical and complex cases.\n",
    "\n",
    "```{figure} resources/intro-fluid-bifurcation.jpg\n",
    "---\n",
    "height: 240px\n",
    "name: intro-fluid-bifurcation \n",
    "---\n",
    "A bifurcation in a buoyancy-driven fluid flow: the \"smoke\" shown in green color starts rising in a percectly straight manner, but tiny numerical inaccuracies grow over time to lead to an instability with vortices alternating to one side (top-right), or in the opposite direction (bottom-right). \n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "useful-special",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "For each of the following notebooks, there's a \"next steps\" section like the one below which contains recommendations about where to start modifying the code. After all, the whole point of these notebooks is to have readily executable programs as a basis for own experiments. The data set and NN sizes of the examples are often quite small to reduce the runtime of the notebooks, but they're nonetheless good starting points for potentially complex and large projects.\n",
    "\n",
    "For the simple DP example above:\n",
    "\n",
    "- This notebook is intentionally using a very simple setup. Change the training setup and NN above to obtain a higher-quality solution such as the green one shown in the very first image at the top. \n",
    "\n",
    "- Or try extending the setup to a 2D case, i.e. a paraboloid. Given the function $\\mathcal P:(y_1,y_2)\\to y_1^2+y_2^2$, find an inverse function $f$ such that $\\mathcal P(f(x)) = x$ for all $x$ in $[0,1]$.\n",
    "\n",
    "- If you want to experiment without installing anything, you can also [[run this notebook in colab]](https://colab.research.google.com/github/tum-pbs/pbdl-book/blob/main/intro-teaser.ipynb). E.g., to try the things mentioned right above."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
