started figures

_toc.yml

@@ -24,6 +24,10 @@
     - file: diffphys-code-sol.ipynb
     - file: diffphys-dpvspinn.md
     - file: diffphys-outlook.md
+- file: physgrad
+  sections:
+    - file: physgrad-comparison.ipynb
+    - file: physgrad-discuss.md
 - file: old-phiflow1.md
   sections:
     - file: overview-burgers-forw-v1.ipynb

diffphys.md

@@ -95,8 +95,6 @@ this would cause huge memory overheads and unnecessarily slow down training.
 Instead, for backpropagation, we can provide faster operations that compute products
 with the Jacobian transpose because we always have a scalar loss function at the end of the chain.
 
-**[TODO check transpose of Jacobians in equations]**
-
 Given the formulation above, we need to resolve the derivatives
 of the chain of function compositions of the $\mathcal P_i$ at some current state $\mathbf{u}^n$ via the chain rule.
 E.g., for two of them

intro.md

@@ -86,6 +86,7 @@ See also... Test link: {doc}`supervised`
 
 - general motivation: repeated solves in classical solvers -> potential for ML
 - PINNs: often need weighting of added loss terms for different parts
+- DP intro, check transpose of Jacobians in equations
 
 
 ## TODOs , Planned content

overview.md

@@ -36,7 +36,7 @@ natural language processing {cite}`radford2019language`,
 and more recently also for protein folding {cite}`alquraishi2019alphafold`.
 The field is very vibrant, and quickly developing, with the promise of vast possibilities.
 
-At the same time, the successes of deep learning (DL) approaches 
+On the other hand, the successes of deep learning (DL) approaches 
 has given rise to concerns that this technology has 
 the potential to replace the traditional, simulation-driven approach to
 science. Instead of relying on models that are carefully crafted
@@ -49,8 +49,8 @@ and _deep learning_.
 One central reason for the importance of this combination is
 that DL approaches are simply not powerful enough by themselves.
 Given the current state of the art, the clear breakthroughs of DL
-in physical applications are outstanding, the proposed techniques are novel,
-sometimes difficult to apply, and
+in physical applications are outstanding. 
+The proposed techniques are novel, sometimes difficult to apply, and
 significant practical difficulties combing physics and DL persist.
 Also, many fundamental theoretical questions remain unaddressed, most importantly
 regarding data efficienty and generalization.
@@ -78,6 +78,16 @@ at our disposal, and use them wherever we can.
 I.e., our goal is to _reconcile_ the data-centered
 viewpoint and the physical simuation viewpoint.
 
+The resulting methods have a huge potential to improve
+what can be done with numerical methods: e.g., in scenarios
+where solves target cases from a certain well-defined problem
+domain repeatedly, it can make a lot of sense to once invest 
+significant resources to train
+an neural network that supports the repeated solves. Based on the
+domain-specific specialization of this network, such a hybrid
+could vastly outperform traditional, generic solvers. And despite
+the many open questions, first publications have demonstrated
+that this goal is not overly far away. 
 
 
 ## Categorization
@@ -134,10 +144,10 @@ each of the different techniques is particularly useful.
 
 To be a bit more specific, _physics_ is a huge field, and we can't cover everything... 
 
-```{note} The focus of this book is on...
+```{note} The focus of this book lies on...
 - _Field-based simulations_ (no Lagrangian methods)
 - Combinations with _deep learning_ (plenty of other interesting ML techniques, but not here)
-- Experiments as _outlook_ (replace synthetic data with real)
+- Experiments as _outlook_ (replace synthetic data with real-world observations)
 ```
 
 It's also worth noting that we're starting to build the methods from some very

supervised.md

@@ -13,20 +13,20 @@ model equations exist.
 ## Problem Setting
 
 For supervised training, we're faced with an 
-unknown function $f^*(x)=y$, collect lots of pairs of data $[x_0,y_0], ...[x_n,y_n]$ (the training data set)
+unknown function $f^*(x)=y^*$, collect lots of pairs of data $[x_0,y^*_0], ...[x_n,y^*_n]$ (the training data set)
 and directly train a NN to represent an approximation of $f^*$ denoted as $f$, such
-that $f(x)=y$.
+that $f(x)=y \approx y^*$.
 
 The $f$ we can obtain is typically not exact, 
 but instead we obtain it via a minimization problem:
 by adjusting weights $\theta$ of our representation with $f$ such that
 
-$\text{arg min}_{\theta} \sum_i (f(x_i ; \theta)-y_i)^2$.
+$\text{arg min}_{\theta} \sum_i (f(x_i ; \theta)-y^*_i)^2$.
 
 This will give us $\theta$ such that $f(x;\theta) \approx y$ as accurately as possible given
 our choice of $f$ and the hyperparameters for training. Note that above we've assumed 
 the simplest case of an $L^2$ loss. A more general version would use an error metric $e(x,y)$
-to be minimized via $\text{arg min}_{\theta} \sum_i e( f(x_i ; \theta) , y_i) )$. The choice
+to be minimized via $\text{arg min}_{\theta} \sum_i e( f(x_i ; \theta) , y^*_i) )$. The choice
 of a suitable metric is topic we will get back to later on.
 
 Irrespective of our choice of metric, this formulation
@@ -47,12 +47,13 @@ which need to be kept small enough for a chosen application. As these topics are
 for classical simulations, the existing knowledge can likewise be leveraged to
 set up DL training tasks.
 
-```{figure} resources/placeholder.png
+```{figure} resources/supervised-training.jpg
 ---
 height: 220px
 name: supervised-training
 ---
-TODO, visual overview of supervised training
+A visual overview of supervised training. Quite simple overall, but it's good to keep this
+in mind in comparison to the more complex variants we'll encounter later on.
 ```
 
 ## Show me some code!