daviddoji/pbdl-book
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

diffphys ns phiflow2 update, WIP

Browse Source
This commit is contained in:
NT 2021-02-27 14:07:42 +08:00
parent feb1477391
commit 6b350893fc
8 changed files with 1069 additions and 374 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
_toc.yml
View File

@ -20,16 +20,19 @@
sections:
- file: diffphys-code-gradient.ipynb
- file: diffphys-discuss.md
- file: diffphys-code-ns-v2a.ipynb
- file: diffphys-code-ns.ipynb
- file: diffphys-code-sol.ipynb
- file: diffphys-dpvspinn.md
- file: diffphys-outlook.md
- file: old-phiflow1.md
sections:
- file: overview-burgers-forw-v1.ipynb
- file: overview-ns-forw-v1.ipynb
- file: diffphys-code-ns.ipynb
- file: physicalloss-code-v2.ipynb
- file: diffphys-code-gradient-v1.ipynb
- file: diffphys-code-tf.ipynb
- file: diffphys-code-ns-v1.ipynb
- file: diffphys-code-ns-v2a.ipynb
- file: jupyter-book-reference
sections:
- file: jupyter-book-reference-markdown

706
diffphys-code-ns.ipynb
View File

File diff suppressed because one or more lines are too long

44
diffphys-discuss.md
View File

@ -1,43 +1,17 @@
Discussion
=======================
In the previous sections we've seen example reconstuctions that used physical residuals as soft constraints, in the form of the PINNs, and reconstuctions that used a differentiable physics (DP) solver. While both methods can find minimizers for the same minimization problem, the solutions the obtained differ substantially, as do the behavior of the non-linear optimization problem that we get from each formulation. In the following we discuss these differences in more detail, and we will combine conclusions drawn from the behavior of the Burgers case of the previous sections with observations from research papers.
## Compatibility with Existing Numerical Methods
It is very obvious that the PINN implementation is quite simple, which is a positive aspect, but at the same time it differs strongly from "typical" discretziations and solution approaches that are usually to employed equations like Burgers equation. The derivatives are computed via the neural network, and hence rely on a fairly accurate representation of the solution to provide a good direction for optimizaion problems.
The DP version on the other hand inherently relies on a numerical solver that is tied into the learning process. As such it requires a discretization of the problem at hand, and given this discretization can employ existing, and potentially powerful numerical techniques. This means solutions and derivatives can be evaluated with known and controllable accuracy, and can be evaluated efficiently.
## Discretization
The reliance on a suitable discretization requires some understanding and knowledge of the problem under consideration. A sub-optimal discretization can impede the learning prcoess or, worst case, lead to diverging trainig runs. However, given the large body of theory and practical realizations of stable solvers for a wide variety of physical problems, this is typically not an unsurmountable obstacle.
The PINN approaches on the other hand do not require an a-priori choice of a discretization, and as such seem to be "discretization less". This, however, is only an advantage on first sight. As they yield solutions in a computer, they naturally _have_ to discretize the problem, but they construct this discretization over the coure of the training process, in a way that is not easily controllable from the outside. Thus, the resulting accuracy is determined by how well the training manages to estimate the complexity of the problem for realistic use cases, and how well the training data approximates the unknown regions of the solution.
As demonstrated with the Burgers example, the PINN solutions typically have significant difficulties propagating information _backward_ in time. This is closely coupled to the efficiency of the method.
## Efficiency
The PINN approach typically perform a localized sampling and correction of the solutions, which means the corrections in the form of weight updates are likewise typically local. The fulfilment of boundary conditions in space and time can be correspondingly slow, leading to long training runs in practice.
A well-chosen discretization of a DP approach can remedy this behavior, and provide an improved flow of gradient information. At the same time, the reliance on a computational grid means that solutions can be obtained very quickly. Given an interpolation scheme or set of basis functions, the solution can be sampled at any point in space or time given a very local neighborhood of the computational grid. Worst case, this can lead to slight memory overheads, e.g., by repeatedly storing mostly constand values of a solution.
For the PINN representation with fully-connected networks on the other hand, we need to make a full pass over the potentially large number of values in the whole network to obtain a sample of the solution at a single point. The network effectively needs to encode the full high-dimensional solution. Its size likewise determines the efficiency of derivative calculations.
**TODO** discussion of DP only
## Summary
The following table summarizes these findings:
Thus, to summarize ...
| Method | ✅ Pro | ❌ Con |
|----------|-------------|------------|
| **PINN** | - Analytic derivatives via back-propagation | - Expensive evaluation of NN, as well as derivative calculations |
| | - Simple to implement | - Incompatible with existing numerical methods |
| | | - No control of discretization |
| **DiffPhys** | - Leverage existing numerical methods | - More complicated to implement |
| | - Efficient evaluation of simulation and derivatives | - Require understanding of problem to choose suitable discretization |
As a summary, both methods are definitely interesting, and leave a lot of room for improvement with more complicated extensions and algorithmic modifications that change and improve on the various negative aspects we have discussed for both sides.
However, as of this writing, the physics-informed (PI) approach has clear limitations when it comes to performance and campatibility with existing numerical methods. Thus, when knowledge of the problem at hand is available, which typically is the case when we choose a suitable PDE model to constrain the learning process, employing a differentiable physics (DP) solver can significantly improve the training process as well as the quality of the obtained solution. Next, we will target more complex settings, i.e., fluids with Navier Stokes, to illustrate this in more detail.
✅ Pro:
- uses physical model and numerical methods for discretization
- efficiency of selected methods carries over to training
- tight coupling of physical models and NNs possible
❌ Con:
- not compatible with all simulators (need to provide gradients)
- ...

44
diffphys-dpvspinn.md Normal file
View File

@ -0,0 +1,44 @@
Diff. Physics versus Phys.-informed Training
=======================
In the previous sections we've seen example reconstuctions that used physical residuals as soft constraints, in the form of the PINNs, and reconstuctions that used a differentiable physics (DP) solver. While both methods can find minimizers for the same minimization problem, the solutions the obtained differ substantially, as do the behavior of the non-linear optimization problem that we get from each formulation. In the following we discuss these differences in more detail, and we will combine conclusions drawn from the behavior of the Burgers case of the previous sections with observations from research papers.
## Compatibility with Existing Numerical Methods
It is very obvious that the PINN implementation is quite simple, which is a positive aspect, but at the same time it differs strongly from "typical" discretziations and solution approaches that are usually to employed equations like Burgers equation. The derivatives are computed via the neural network, and hence rely on a fairly accurate representation of the solution to provide a good direction for optimizaion problems.
The DP version on the other hand inherently relies on a numerical solver that is tied into the learning process. As such it requires a discretization of the problem at hand, and given this discretization can employ existing, and potentially powerful numerical techniques. This means solutions and derivatives can be evaluated with known and controllable accuracy, and can be evaluated efficiently.
## Discretization
The reliance on a suitable discretization requires some understanding and knowledge of the problem under consideration. A sub-optimal discretization can impede the learning prcoess or, worst case, lead to diverging trainig runs. However, given the large body of theory and practical realizations of stable solvers for a wide variety of physical problems, this is typically not an unsurmountable obstacle.
The PINN approaches on the other hand do not require an a-priori choice of a discretization, and as such seem to be "discretization less". This, however, is only an advantage on first sight. As they yield solutions in a computer, they naturally _have_ to discretize the problem, but they construct this discretization over the coure of the training process, in a way that is not easily controllable from the outside. Thus, the resulting accuracy is determined by how well the training manages to estimate the complexity of the problem for realistic use cases, and how well the training data approximates the unknown regions of the solution.
As demonstrated with the Burgers example, the PINN solutions typically have significant difficulties propagating information _backward_ in time. This is closely coupled to the efficiency of the method.
## Efficiency
The PINN approach typically perform a localized sampling and correction of the solutions, which means the corrections in the form of weight updates are likewise typically local. The fulfilment of boundary conditions in space and time can be correspondingly slow, leading to long training runs in practice.
A well-chosen discretization of a DP approach can remedy this behavior, and provide an improved flow of gradient information. At the same time, the reliance on a computational grid means that solutions can be obtained very quickly. Given an interpolation scheme or set of basis functions, the solution can be sampled at any point in space or time given a very local neighborhood of the computational grid. Worst case, this can lead to slight memory overheads, e.g., by repeatedly storing mostly constand values of a solution.
For the PINN representation with fully-connected networks on the other hand, we need to make a full pass over the potentially large number of values in the whole network to obtain a sample of the solution at a single point. The network effectively needs to encode the full high-dimensional solution. Its size likewise determines the efficiency of derivative calculations.
## Summary
The following table summarizes these findings:
| Method | ✅ Pro | ❌ Con |
|----------|-------------|------------|
| **PINN** | - Analytic derivatives via back-propagation | - Expensive evaluation of NN, as well as derivative calculations |
| | - Simple to implement | - Incompatible with existing numerical methods |
| | | - No control of discretization |
| **DiffPhys** | - Leverage existing numerical methods | - More complicated to implement |
| | - Efficient evaluation of simulation and derivatives | - Require understanding of problem to choose suitable discretization |
As a summary, both methods are definitely interesting, and have a lot of potential. There are numerous more complicated extensions and algorithmic modifications that change and improve on the various negative aspects we have discussed for both sides.
However, as of this writing, the physics-informed (PI) approach has clear limitations when it comes to performance and campatibility with existing numerical methods. Thus, when knowledge of the problem at hand is available, which typically is the case when we choose a suitable PDE model to constrain the learning process, employing a differentiable physics (DP) solver can significantly improve the training process as well as the quality of the obtained solution. Next, we will target more complex settings, i.e., fluids with Navier Stokes, to illustrate this in more detail.

2
overview.md
View File

@ -150,7 +150,7 @@ fundamental steps. Here are some considerations for skipping ahead to the later
- On the other hand, if you're already deep into ANNs&Co, and you'd like to skip ahead to the research related topics, we recommend starting in the _Physical Loss Terms_ chapter, which lays the foundations for the next chapters.
A brief look at our _Notation_ won't hurt in both cases, though!
A brief look at our _notation_ in the {doc}`notation` chapter won't hurt in both cases, though!
```
---

600
physicalloss-code-v2.ipynb Normal file
View File

File diff suppressed because one or more lines are too long

32
physicalloss-code.ipynb
View File

@ -6,7 +6,7 @@
"source": [
"# Burgers Optimization with a Physics-Informed NN\n",
"\n",
"Warning: this example still needs phiflow1.x and tensorflow 1.x! \n",
"**Warning: this example still needs phiflow1.x and tensorflow 1.x! **\n",
"\n",
"To illustrate how the physics-informed losses work, let's consider a \n",
"reconstruction example with a simple yet non-linear equation in 1D:\n",
@ -25,33 +25,9 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Could not load resample cuda libraries: CUDA binaries not found at /Users/thuerey/Dropbox/mbaDevelSelected/phiflow-v1.5/phi/tf/cuda/build/resample.so. Run \"python setup.py cuda\" to compile them\n",
"WARNING:tensorflow:From /Users/thuerey/Dropbox/mbaDevelSelected/phiflow-v1.5/phi/tf/util.py:119: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.\n",
"\n",
"WARNING:tensorflow:From /Users/thuerey/Dropbox/mbaDevelSelected/phiflow-v1.5/phi/tf/profiling.py:12: The name tf.RunOptions is deprecated. Please use tf.compat.v1.RunOptions instead.\n",
"\n",
"WARNING:tensorflow:From /Users/thuerey/Dropbox/mbaDevelSelected/phiflow-v1.5/phi/tf/profiling.py:13: The name tf.RunMetadata is deprecated. Please use tf.compat.v1.RunMetadata instead.\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/thuerey/Dropbox/mbaDevelSelected/phiflow-v1.5/phi/tf/flow.py:15: UserWarning: TensorFlow-CUDA solver is not available. To compile it, download phiflow sources and run\n",
"$ python setup.py tf_cuda\n",
"before reinstalling phiflow.\n",
" warnings.warn(\"TensorFlow-CUDA solver is not available. To compile it, download phiflow sources and run\\n$ python setup.py tf_cuda\\nbefore reinstalling phiflow.\")\n"
]
}
],
"outputs": [],
"source": [
"from phi.tf.flow import *\n",
"import numpy as np\n",
@ -777,7 +753,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
"version": "3.8.5"
}
},
"nbformat": 4,

8
physicalloss-discuss.md
View File

@ -58,13 +58,13 @@ goals of the next sections.
✅ Pro:
- uses physical model
- derivatives via backpropagation
- derivatives can be convieniently compute via backpropagation
❌ Con:
- slow ...
- only soft constraints
- quite slow ...
- physical constraints are enforced only as soft constraints
- largely incompatible _classical_ numerical methods
- derivatives rely on learned representation
- accuracy of derivatives relies on learned representation
Next, let's look at how we can leverage numerical methods to improve the DL accuracy and efficiency
by making use of differentiable solvers.