cleanup, added other topics chapter

_toc.yml

@@ -49,6 +49,14 @@
   - file: bayesian-intro.md
   - file: bayesian-code.md
   
+- part: Fast Forward Topics
+  chapters:
+  - file: others-intro.md
+  - file: others-timeseries.md
+  - file: others-GANs.md
+  - file: others-lagrangian.md
+  - file: others-metrics.md
+  
 - part: End Matter
   chapters:
   - file: outlook.md

bayesian-intro.md

@@ -1,14 +1,31 @@
 Introduction to Posterior Inference
 =======================
 
-more complex models -> fit data better , but overfit & dont generalize
--> posterior (distrib) inference
+We have to keep in mind that for all measurements, models, and discretizations we have uncertainties. In the former, this typically appears in the form of measurements errors, model equations usually encompass only parts of a system we're interested in, and for numerical simulations we inherently introduce discretization errors. So a very important question to ask here is how sure we can be sure that an answer we obtain is the correct one. From a statistics viewpoint, we'd like to know the probability distribution for the posterior, i.e., the outcomes.
 
-std NNs do MLE for weights
-"theoretically justifiable" ie correct thing to do -> "posterior inference"
-extremely difficult
+This admittedly becomes even more difficult in the context of machine learning:
+we're typically facing the task of approximating complex and unknown functions.
+From a probabilistic perspective, the standard process of training an NN here
+yields a _maximum likelihood estimation_ (MLE) for the parameters of the network.
+However, this MLE viewpoint does not take any of the uncertainties mentioned above into account:
+for DL training, we likewise have a numerical optimization, and hence an inherent
+approximation error and uncertainty regarding the learned representation.
+Ideally, we could change our learning problem such that we could do _posterior inference_,
+i.e. learn to produce the full output distribution. However, this turns out to be an
+extremely difficult task.
 
-useful for uncertainty! esp interesting in phys setting
+This where so called _Bayesian neural network_ (BNN) approaches come into play. They 
+make posterior inference possible by making assumptions about the probability 
+distributions of individual parameters of the network. Nonetheless, the task
+remains very challenging. Training a BNN is typically significantly more difficult
+than training a regular NN. However, this should come as no surprise, as we're trying to 
+learn something fundamentally different in this case: a full probability distribution 
+instead of a point estimate.
+
+![Divider](resources/divider5.jpg)
+
+## A practical example
 
 first example here with airfoils, extension from {doc}`supervised-airfoils`
 
+

diffphys-control.ipynb

@@ -536,18 +536,6 @@
           ],
           "name": "stdout"
         },
-        {
-          "output_type": "stream",
-          "text": [
-            "/usr/local/lib/python3.7/dist-packages/phi/struct/context.py:18: UserWarning: struct.unsafe() is deprecated. Use map() with new_type argument to avoid validation.\n",
-            "  warnings.warn(\"struct.unsafe() is deprecated. Use map() with new_type argument to avoid validation.\")\n",
-            "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:414: UserWarning: `tf.layers.conv2d` is deprecated and will be removed in a future version. Please Use `tf.keras.layers.Conv2D` instead.\n",
-            "  warnings.warn('`tf.layers.conv2d` is deprecated and '\n",
-            "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py:1719: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.\n",
-            "  warnings.warn('`layer.apply` is deprecated and '\n"
-          ],
-          "name": "stderr"
-        },
         {
           "output_type": "stream",
           "text": [
@@ -783,4 +771,4 @@
       "outputs": []
     }
   ]
-}
+}

diffphys-dpvspinn.md

@@ -1,4 +1,4 @@
-Diff. Physics versus Phys.-informed Training
+Differentiable Physics versus Physics-informed Training
 =======================
 
 In the previous sections we've seen example reconstructions that used physical residuals as soft constraints, in the form of the PINNs, and reconstructions that used a differentiable physics (DP) solver. While both methods can find minimizers for the same minimization problem, the solutions the obtained differ substantially, as do the behavior of the non-linear optimization problem that we get from each formulation. In the following we discuss these differences in more detail, and we will combine conclusions drawn from the behavior of the Burgers case of the previous sections with observations from research papers.

others-GANs.md

@@ -0,0 +1,6 @@
+Generative Adversarial Networks
+=======================
+
+...
+
+

others-intro.md

@@ -0,0 +1,19 @@
+Additional Topics
+=======================
+
+The next sections will give a shorter introduction to other topics that are highly 
+interesting in the context of physics-based deep learning.
+
+![Divider](resources/divider4.jpg)
+
+More specifically, we will look at:
+
+* Time series predictions, i.e., using to DL predict the evolution of a physical system.	
+  This typically replaces a numerical solver, and we can make use of special techniques from the DL area that target time series.
+
+* Generative models are likewise an own topic in DL, and here especially generative adversarial networks were shown to be powerful tools. They also represent a highly interesting training approach involving to separate NNs.
+
+* Meshless methods and unstructured meshes are an important topic for classical simulations. Here, we'll look at a specific Lagrangian method that employs learning in the context of dynamic, particle-based representations.
+
+* Finally, metrics to reboustly assess the quality of similarity of measurements and results are a central topic for all numerical methods, no matter whether they employ learning or not. In the last section we will look at how DL can be used to learn specialized and improved metrics.
+

others-lagrangian.md

@@ -0,0 +1,6 @@
+Meshless Methods
+=======================
+
+...
+
+

others-metrics.md

@@ -0,0 +1,6 @@
+Metric Learning
+=======================
+
+...
+
+

others-timeseries.md

@@ -0,0 +1,7 @@
+Model Reduction and Time Series
+=======================
+
+model reduction? separate
+
+...
+

overview-equations.md

@@ -139,7 +139,9 @@ with actual simulations and implementation examples on the next page.
 
 ---
 
-## Some PDEs that we'll use later on
+## Some example PDEs 
+
+The following PDEs are good examples, and we'll use them later on in different settings to show how to incorporate them into DL approaches.
 
 ### Burgers
 
@@ -167,7 +169,7 @@ $\frac{\partial u}{\partial{t}} + u \nabla u = \nu \nabla \cdot \nabla u $ .
 
 ### Navier-Stokes
 
-An interesting next step in terms of complexity is given by the
+A good next step in terms of complexity is given by the
 Navier-Stokes equations, which are a well-established model for fluids.
 In addition to an equation for the conservation of momentum (similar to Burgers),
 they include an equation for the conservation of mass. This prevents the 
@@ -205,7 +207,7 @@ $\begin{aligned}
 
 where $\xi$ denotes the strength of the buoyancy force.
 
-And finally, we'll also consider 3D cases with the Navier-Stokes model, i.e.:
+And finally, the Navier-Stokes model in 3D give the following set of equations:
 
 $
 \begin{aligned}
@@ -219,6 +221,6 @@ $
 \end{aligned}
 $
 
-### More to come...
+## Forward Simulations
 
-In the future, we'll include some other model equations as well. Stay tuned...
+Before we really start with learning methods, it's important to cover the most basic variant of using the above model equations: a regular "forward" simulation, that starts from a set of initial conditions, and evolves the state of the system over time with a discretized version of the model equation. We'll show how to run such forward simulations for Burgers' equation in 1D and for a 2D Navier-Stokes simulation.

physgrad.md

@@ -1,9 +1,6 @@
 Physical Gradients
 =======================
 
-**TODO, finish training section below? integrate comment at bottom?**
-**relate to invertible NNs?**
-
 The next chapter will dive deeper into state-of-the-art-research, and aim for an even tighter
 integration of physics and learning.
 The approaches explained previously all integrate physical models into deep learning algorithms,

physicalloss-discuss.md

@@ -19,7 +19,7 @@ when it doesn't pick the right regions of the solution.
 ## Is it "Machine Learning"?
 
 One question that might also come to mind at this point is: _can we really call it machine learning_?
-Of course, such denomination questions are mostly superficial - if an algorithm is useful, it doesn't matter
+Of course, such denomination questions are superficial - if an algorithm is useful, it doesn't matter
 what name it has. However, here the question helps to highlight some important properties
 that are typically associated with algorithms from fields like machine learning or optimization.
 

supervised-airfoils.ipynb

@@ -935,8 +935,7 @@
     "* Experiment with learning rate, dropout, and model size to reduce the error on the test set. How small can you make it with the given training data?\n",
     "\n",
     "* As you'll see, it's a bit limited here what you can get out of this dataset, head over to [the main github repo of this project](https://github.com/thunil/Deep-Flow-Prediction) to download larger data sets, or generate own data\n",
-    "\n",
-    "**(TODO, us: for exercise, provide data with \"errors\" (nan & huge neg number in 1 cell), filter out to make model train...)**\n"
+    "\n"
    ]
   },
   {