updated intro teaser
This commit is contained in:
N_T committed
1 parent
39fcd963ab
commit
b7667370d2
5 files changed
+411
-116
No files matched your search
+411
-116
@@ -13,16 +13,16 @@
|
||||
"id": "original-brave",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's start with a very reduced example that highlights some of the key capabilities of physics-based learning approaches. Let's assume our physical model is a very simple equation: a parabola along the positive x-axis.\n",
|
||||
"Let's start with a very reduced example that highlights some of the key capabilities of physics-based learning approaches. Let's assume our physical model is a very simple equation: a parabola along the positive x-axis. We'll also directly use this example to give an outlook towards probabilistic \"generative AI\" approaches.\n",
|
||||
"\n",
|
||||
"Despite being very simple, there are two solutions for every point along x, i.e. we have two modes, one above the other one below the x-axis, as shown on the left below. If we don't take care, a conventional learning approach will give us an approximation like the red one shown in the middle, which is completely off. With an improved learning setup, ideally, by using a discretized numerical solver, we can at least accurately represent one of the modes of the solution (shown in green on the right).\n",
|
||||
"Despite being very simple, there are two solutions for every point along x, i.e. we have two _modes_, one above the other one below the x-axis, as shown on the left below. If we don't take care, a conventional learning approach will give us an approximation like the red one shown in the middle, which is completely off. With an improved learning setup, e.g., by using a discretized numerical solver, we can at least accurately represent one of the modes of the solution (shown in green on the right). Interestingly, approaches that learn the full distribution at each point, flow matching as a representative of diffusion models is used below, can capture both modes!\n",
|
||||
"\n",
|
||||
"```{figure} resources/intro-teaser-side-by-side.png\n",
|
||||
"```{figure} resources/intro-teaser-side-by-side.jpg\n",
|
||||
"---\n",
|
||||
"height: 180px\n",
|
||||
"height: 120px\n",
|
||||
"name: intro-teaser-side-by-side\n",
|
||||
"---\n",
|
||||
"Side by side - supervised versus differentiable physics training.\n",
|
||||
"Side by side - supervised versus differentiable physics and probabilistic training.\n",
|
||||
"```\n"
|
||||
]
|
||||
},
|
||||
@@ -41,15 +41,15 @@
|
||||
"source": [
|
||||
"One of the key concepts of the following chapters is what we'll call _differentiable physics_ (DP). This means that we use domain knowledge in the form of model equations, and then integrate discretized versions of these models into the training process. As implied by the name, having differentiable formulations and operators is crucial for this process to integrate with neural networks training.\n",
|
||||
"\n",
|
||||
"Let's illustrate the properties of deep learning via DP with the following example: We'd like to find an unknown function $f^*$ that generates solutions from a space $Y$, taking inputs from $X$, i.e. $f^*: X \\to Y$. In the following, we'll often denote _idealized_, and unknown functions with a $*$ superscript, in contrast to their discretized, realizable counterparts without this superscript. \n",
|
||||
"\n",
|
||||
"Let's additionally assume we have a generic differential equation $\\mathcal P^*: Y \\to Z$ (our _model_ equation), that encodes a property of the solutions, e.g. some real world behavior we'd like to match. Later on, $P^*$ will often represent time evolutions, but it could also be a conservation law (e.g., conservation of mass, then $\\mathcal P^*$ would measure divergence). \n",
|
||||
"Let's illustrate the properties of deep learning via DP with the following example: We'd like to find an unknown function $f^*$ that generates solutions from a space $Y$, taking inputs from $X$, i.e. $f^*: X \\to Y$. In the following, we'll often denote _idealized_, and unknown functions with a $*$ superscript, in contrast to their discretized, realizable counterparts without this superscript. Let's additionally assume we have a generic differential equation $\\mathcal P^*: Y \\to Z$ (our _model_ equation), that encodes a property of the solutions, e.g. some real world behavior we'd like to match. Later on, $P^*$ will often represent time evolutions, but it could also be a conservation law (e.g., conservation of mass, then $\\mathcal P^*$ would measure divergence). \n",
|
||||
"\n",
|
||||
"Using a neural network $f$ to learn the unknown and ideal function $f^*$, we could turn to classic _supervised_ training to obtain $f$ by collecting data. This classical setup requires a dataset by sampling $x$ from $X$ and adding the corresponding solutions $y$ from $Y$. We could obtain these, e.g., by classical numerical techniques. Then we train the NN $f$ with classic methods using this dataset. \n",
|
||||
"\n",
|
||||
"In contrast to this supervised approach, employing a differentiable physics approach takes advantage of the fact that we can often use a discretized version of the physical model $\\mathcal P$ and employ it to guide the training of $f$. I.e., we want $f$ to be aware of our _simulator_ $\\mathcal P$, and to _interact_ with it. This can vastly improve the learning, as we'll illustrate below with a very simple example (more complex ones will follow later on).\n",
|
||||
"In contrast to this supervised approach, employing a differentiable physics approach takes advantage of the fact that we can often use a discretized version of the physical model $\\mathcal P$ and employ it to guide the training of $f$. I.e., we want $f$ to be aware of our _simulator_ $\\mathcal P$, and to _interact_ with it. This can give fundamental improvements, as we'll illustrate below with a very simple example (more complex ones will follow later on).\n",
|
||||
"\n",
|
||||
"Note that in order for the DP approach to work, $\\mathcal P$ has to be differentiable, as implied by the name. These differentials, in the form of a gradient, are what's driving the learning process.\n"
|
||||
"Note that in order for the DP approach to work, $\\mathcal P$ has to be _differentiable_, as implied by the name. These differentials, in the form of a gradient, are what's driving the learning process and neural network integration.\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -81,8 +81,9 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import tensorflow as tf\n",
|
||||
"import matplotlib.pyplot as plt"
|
||||
"import torch\n",
|
||||
"import torch.nn as nn\n",
|
||||
"import torch.optim as optim"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -100,21 +101,17 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# X-Data\n",
|
||||
"N = 200\n",
|
||||
"X = np.random.random(N)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "capable-month",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Generate data\n",
|
||||
"N = 10000\n",
|
||||
"X = np.random.random(N).astype(np.float32).reshape(-1, 1)\n",
|
||||
"\n",
|
||||
"# Generation of Y-Data\n",
|
||||
"sign = (- np.ones((N,)))**np.random.randint(2,size=N)\n",
|
||||
"Y = np.sqrt(X) * sign"
|
||||
"sign = (- np.ones((N,))).astype(np.float32) ** np.random.randint(2, size=N)\n",
|
||||
"Y = (np.sqrt(X.flatten()) * sign).reshape(-1, 1).astype(np.float32)\n",
|
||||
"\n",
|
||||
"# Convert to PyTorch tensors\n",
|
||||
"X_tensor = torch.tensor(X)\n",
|
||||
"Y_tensor = torch.tensor(Y)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -122,22 +119,30 @@
|
||||
"id": "stone-science",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now we can define a network, the loss, and the training configuration. We'll use a simple `keras` architecture with three hidden layers and ReLU activations."
|
||||
"Now we can define a network. We'll use a simple fully connected architecture with three hidden layers and ReLU activations."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"execution_count": 3,
|
||||
"id": "weighted-costa",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Neural network\n",
|
||||
"act = tf.keras.layers.ReLU()\n",
|
||||
"nn_sv = tf.keras.models.Sequential([\n",
|
||||
" tf.keras.layers.Dense(10, activation=act, input_shape=(1,)),\n",
|
||||
" tf.keras.layers.Dense(10, activation=act),\n",
|
||||
" tf.keras.layers.Dense(1,activation='linear')])"
|
||||
"# Define the neural network\n",
|
||||
"class SimpleNN(nn.Module):\n",
|
||||
" def __init__(self, hiddendim=10):\n",
|
||||
" super(SimpleNN, self).__init__()\n",
|
||||
" self.fc1 = nn.Linear(1, hiddendim)\n",
|
||||
" self.fc2 = nn.Linear(hiddendim, hiddendim)\n",
|
||||
" self.fc3 = nn.Linear(hiddendim, 1)\n",
|
||||
" self.relu = nn.ReLU()\n",
|
||||
" \n",
|
||||
" def forward(self, x):\n",
|
||||
" x = self.relu(self.fc1(x))\n",
|
||||
" x = self.relu(self.fc2(x))\n",
|
||||
" x = self.fc3(x) # Linear output\n",
|
||||
" return x"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -145,12 +150,12 @@
|
||||
"id": "confirmed-cargo",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"And we can start training via a simple mean squared error loss, using `fit` function from keras:"
|
||||
"Next we can instantiate the model (using a hidden dimension of 128), specify a loss function (will used a simple mean squared error with PyTorch's `MSELoss()`), and the Adam optimizer. The network is trained for 50 epochs in the loop below:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"execution_count": 4,
|
||||
"id": "adolescent-yellow",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
@@ -158,27 +163,40 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 1/5\n",
|
||||
"40/40 [==============================] - 0s 1ms/step - loss: 0.5084\n",
|
||||
"Epoch 2/5\n",
|
||||
"40/40 [==============================] - 0s 1ms/step - loss: 0.5022\n",
|
||||
"Epoch 3/5\n",
|
||||
"40/40 [==============================] - 0s 1ms/step - loss: 0.5011\n",
|
||||
"Epoch 4/5\n",
|
||||
"40/40 [==============================] - 0s 1ms/step - loss: 0.5002\n",
|
||||
"Epoch 5/5\n",
|
||||
"40/40 [==============================] - 0s 1ms/step - loss: 0.5007\n"
|
||||
"Epoch 10/50, Loss: 0.100193\n",
|
||||
"Epoch 20/50, Loss: 0.100193\n",
|
||||
"Epoch 30/50, Loss: 0.100207\n",
|
||||
"Epoch 40/50, Loss: 0.100202\n",
|
||||
"Epoch 50/50, Loss: 0.100203\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Loss function\n",
|
||||
"loss_sv = tf.keras.losses.MeanSquaredError()\n",
|
||||
"optimizer_sv = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
|
||||
"nn_sv.compile(optimizer=optimizer_sv, loss=loss_sv)\n",
|
||||
"nn_sup = SimpleNN(hiddendim=128)\n",
|
||||
"criterion = nn.MSELoss()\n",
|
||||
"optimizer = optim.Adam(nn_sup.parameters(), lr=0.001)\n",
|
||||
"\n",
|
||||
"# Training\n",
|
||||
"results_sv = nn_sv.fit(X, Y, epochs=5, batch_size= 5, verbose=1)"
|
||||
"# Training loop\n",
|
||||
"epochs = 50\n",
|
||||
"batch_size = 5\n",
|
||||
"\n",
|
||||
"for epoch in range(epochs):\n",
|
||||
" permutation = torch.randperm(N)\n",
|
||||
" epoch_loss = 0.0\n",
|
||||
" \n",
|
||||
" for i in range(0, N, batch_size):\n",
|
||||
" indices = permutation[i:i+batch_size]\n",
|
||||
" batch_x, batch_y = X_tensor[indices], Y_tensor[indices]\n",
|
||||
" \n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" outputs = nn_sup(batch_x)\n",
|
||||
" loss = criterion(outputs, batch_y)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" \n",
|
||||
" epoch_loss += loss.item()\n",
|
||||
" \n",
|
||||
" if(epoch%10==9): print(f\"Epoch {epoch+1}/{epochs}, Loss: {epoch_loss/N:.6f}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -186,32 +204,31 @@
|
||||
"id": "governmental-mixture",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"As both NN and the data set are very small, the training converges very quickly. Let's plot the solution: the following one shows the data in light gray, and the supervised solution in red. "
|
||||
"As both NN and the data set are fairly small, the training converges quickly. Let's plot the solution: the following one shows the data in light gray, and the supervised solution in red. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"execution_count": 5,
|
||||
"id": "sought-basement",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEWCAYAAABIVsEJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2eElEQVR4nO3dfXwddZn//9fVk6Zp2tSmbShNS2+ylJsKWDDcdFEsq9yo/AAX1IIKXQqIbPHrsqL4RVcE0QqrKD9YpQoCKncrClV0AQsFlLLQYuWmCC2lQO/TJqE3h/QmXN8/ZiadTObkPjk5yfv5eJxHzsx8Zs5nTpJznc+9uTsiIiIdNSjfGRARkcKkACIiIp2iACIiIp2iACIiIp2iACIiIp2iACIiIp2iACLSBjObaWZruvF6s83sz911vb6mv9+f7KUAIgXBzD5gZk+Z2dtmVmtmfzGzI8Nj+sASyYOifGdApC1mNgL4PfAF4F6gGPggsDOf+WoPMyty9z35zkeavpw3KQwqgUghOADA3e9y90Z3f8fdH3b3583sYOAnwAwz225m9QBm9nEz+6uZbTWzt8zsyuhiZjbZzNzMzjWzN81ss5ldETs+1MxuM7M6M1sOHBnPjJldbmavmdk2M1tuZp+IHZsdlo6uN7MtwJVmNtrMFoR5eQb4h9Zu1sz+28w2hKWtJ8zsvbFjt5nZT8zskfD1HzezSbHjbmZfNLNV4X1dZ2aDWsnbe8zsDjOrMbM3zOzrsfT/YGaPmtmW8Fq/MrORsdfaz8x+E567xcxuTNzHf4bv4etm9tE2fsdSgBRApBC8CjSa2e1m9lEzK48OuPvLwEXAYncf7u4jw0M7gHOAkcDHgS+Y2emJ634AOBD4MPAfYTAC+CbBh/w/ACcB5ybOe42gBPQe4FvAL81sXOz40cAqYCxwDXAT0ACMA84LH635IzAV2Ad4DvhV4vhngKuBMcCylOOfAKqBI4DTEq+XzNv/H95HFfAhgvfsX8K0BnwXqAQOBvYDrgQwswxBqfANYDIwHrg78TqvhHm8FrjFzKyN+5ZC4+566NHnHwQfYLcBa4A9wAJgbHhsNvDnNs7/IXB9+Hwy4MCE2PFngFnh81XAybFjFwJrWrn2MuC0WF7ejB3LALuBg2L7vtNWfmNpR4Z5fU+4fRtwd+z4cKAR2C/c9kTeLwYWtpK3XcC02L7PA4ty5OV04K/h8xlADVCUkm42sDK2XRrma998/x3p0b0PlUCkILj7y+4+290nAIcQfCv+Ya70Zna0mT0WVq+8TVBKGZNItiH2PEvwYUx47bdix95IXPscM1tmZvVhldkhiWvHz60gaGvMeb3EtTNmNi+sItsKrA4PpV7f3bcDtWGe017/jVaOjQEGJ/LzBkFpAjMba2Z3m9naMC+/jOVjP+ANz92G0vTeuns2fDo8R1opUAogUnDc/e8E38QPiXalJLuToJSyn7u/h6CdpL1VKOsJPiAjE6MnYXvDT4G5wGgPqsxeTFw7np8aghJT6vVSnE1Q7fQRgqqlydFLx9I0XcvMhgOjgHVpx8PXih+L520zQeloUiL92vD5d8L0h7r7COCzsXy8BUw0M3XEGcAUQKTPM7ODzOzfzWxCuL0fcBbwdJhkIzDBzIpjp5UBte7eYGZHEXwwt9e9wNfMrDx8zUtix4YRfKjWhHn5F/YGshbcvRH4DUGDdamZTaNlm0pcGUHvsi0EVT/fSUnzMQu6NRcTtIU87e7xksVlYd73A/4PcE8rebsXuMbMysLgeClBSSPKy3bgbTMbD1wWO/0ZgkA7z8yGmVmJmR3byn1JP6QAIoVgG0Gj7P+a2Q6CwPEi8O/h8UeBl4ANZrY53HcxcJWZbQP+g+CDsr2+RVCV8zrwMPCL6IC7Lwe+DywmCFyHAn9p43pzCapvNhCUnH7eSto7wtdeCyxnb5CMu5Ogob8WeD9BySDuAWApQdvMg8AtrbzeJQQdDlYBfw6vfWt47FsEDfFvh9f5TXRSGHz+P2B/4E2CtqlPt/I60g+ZuxaUEikUZnYbQYP+13Mcd2Cqu6/s1YzJgKQSiIiIdIoCiIiIdIqqsEREpFNUAhERkU4ZUH24x4wZ45MnT853NkRECsrSpUs3u3tFcv+ACiCTJ09myZIl+c6GiEhBMbPU2RNUhSUiIp2iACIiIp2iACIiIp0yoNpA0uzevZs1a9bQ0NCQ76wMCCUlJUyYMIHBgwfnOysi0kV5DSBmditwCrDJ3VtMSBcuQPMj4GME023PdvfnwmPnAtF0Dt9299s7k4c1a9ZQVlbG5MmT0Xo3Pcvd2bJlC2vWrGHKlCn5zo6IdFG+q7BuA05u5fhHCVZmm0qwqM+PAcxsFMFkckcDRwHfjK9S1xENDQ2MHj1awaMXmBmjR49WaU+kn8hrAHH3JwhmFM3lNOAODzwNjAyXDj0JeMTda929DniE1gNRqxQ8eo/ea5Hekc1mqampIZvNtp24k/p6G8h4mq+gtibcl2t/C2Z2IUHphYkTW1vHR0Sk8GWzWerq6qivr8fdMTOmTJlCaWlpt79Wvquwepy7z3f3anevrqhoMZCyT8hkMkyfPp33vve9vO997+P73/8+7777bqvnrF69mjvvvLPH83b++eezfPnyVtPcf//9baYRkc7JZrOsXbuWtWvXsmHDBlavXk1tbXrFTTab5fXXX6euro5onkN3Z8eOHT2St75eAllL8+U5J4T71gIzE/sX9VquutnQoUNZtmwZAJs2beLss89m69atfOtb38p5ThRAzj67IwvtddzPfvazNtPcf//9nHLKKUybNq1H8yIyEGSzWXbs2EEmk+Gdd96hrq6uRZrt27cDMGrUqGb7d+zYQXKCXDNj2LBhPZLXvl4CWQCcY4FjgLfdfT3wEHBiuGxnOXBiuK9X9GTd4j777MP8+fO58cYbcXdWr17NBz/4QY444giOOOIInnrqKQAuv/xynnzySaZPn87111+fM13c6tWrOeigg/jMZz7DwQcfzJlnntl0DwsXLuTwww/n0EMP5bzzzmPnzp0AzJw5s2n6l+HDh3PFFVfwvve9j2OOOYaNGzfy1FNPsWDBAi677DKmT5/Oa6+9xg033MC0adM47LDDmDVrVre/RyL9QfJzJCpprFq1io0bN7Ju3brU4BHZunVri33Dhg1r1s5YXl7eY9VXkP9uvHcRlCTGmNkagp5VgwHc/SfAHwi68K4k6Mb7L+GxWjO7Gng2vNRV7t5aY3y3iYqIPVm3WFVVRWNjI5s2bWKfffbhkUceoaSkhBUrVnDWWWexZMkS5s2bx3/+53/y+9//vilfaemSXnnlFW655RaOPfZYzjvvPP7rv/6LuXPnMnv2bBYuXMgBBxzAOeecw49//GO+9KUvNTt3x44dHHPMMVxzzTV85Stf4ac//Slf//rXOfXUUznllFM488wzAZg3bx6vv/46Q4YMob6+vlvfG5FCFJUqopJAXV1dU3AwM8aNG8f69etblB5aM2LEiBb7SktLmTJlStNr9VTgiOQ1gLj7WW0cd+Bfcxy7lb1rN/eaeBExqlvsyV/S7t27mTt3LsuWLSOTyfDqq692Kd1+++3HscceC8BnP/tZbrjhBk444QSmTJnCAQccAMC5557LTTfd1CKAFBcXc8oppwDw/ve/n0ceeST1NQ477DA+85nPcPrpp3P66ad34q5FCk9tbS1bt25lxIgRzaqW4l86IQgY8UDh7mzdujVn8DAzhg8fTlFREZlMhoaGhhavEVdaWtrjgSPS19tA+pyoiBiVQHqibnHVqlVkMhn22WcfvvWtbzF27Fj+9re/8e6771JSUpJ6zvXXX9+udMlutB3pVjt48OCm9JlMhj179qSme/DBB3niiSf43e9+xzXXXMMLL7xAUZH+1KT/qq2tZd26dUDL9olku0RaG8WIESOa0pkZI0eOZOjQoTQ2NvZKSaKz9F/dQT1dRKypqeGiiy5i7ty5mBlvv/02EyZMYNCgQdx+++00NjYCUFZWxrZt25rOy5Uu6c0332Tx4sXMmDGDO++8kw984AMceOCBrF69mpUrV7L//vvzi1/8gg996EPtznM8L++++y5vvfUWxx9/PB/4wAe4++672b59OyNHjuz8myKSR7lKFnHJ9oitW7c2pY1/6QSafQEdOXIk5eXllJaWUlJS0mtVT91FAaQTuruI+M477zB9+nR2795NUVERn/vc57j00ksBuPjiiznjjDO44447OPnkk5tKPIcddhiZTIb3ve99zJ49O2e6pAMPPJCbbrqJ8847j2nTpvGFL3yBkpISfv7zn/PJT36SPXv2cOSRR3LRRRe1O/+zZs3iggsu4IYbbuDuu+9mzpw5vP3227g7X/ziFxU8pGBEwaKkpIRMJkNjYyObN28Gcvd8gqA9IjoebUeSXzqB1EDRm1VP3WVArYleXV3tyYbll19+mYMPPjhPOepdq1ev5pRTTuHFF1/Maz4G0nsufUeyeyzQrJqooaGhqRoql+HDh5NrVdP2lFQKlZktdffq5H6VQESk34oHjbReTvGeULnaDePSej5FRo0a1e8CR1sUQAaQyZMn5730IdLdamtrqa2tZfDgwVRUVDRVAyV7P7XG3VM7esQnAO2PJYuuUgARkYIRL1E0NjbS0NDA22+/DQQza2/bto2qqipKS0tTR2XnYmZUVFRQVlbWrA2kkBq080EBRET6vGjUdrznYS7R2Kxkl/tx48bR2NiYsw0kasRWKaP9FEBEpE/LZrOsWrWq3emjnk69PSp7IFIAEZFeFZ/Wo6Ghoc2eS+2dSba0tJR999234LvGFhIFkD7gmmuu4c477ySTyTBo0CBuvvlmjj766F7Pxz/+4z+mTsLYEYsWLWo2R5cMTNGaFEDTQLlof1rDdmtjLNLGNI0ZM4ZMJtPUFqISRn4ogOTZ4sWL+f3vf89zzz3HkCFD2Lx5M7t27eqR13J33J1Bg9InYe5q8BCLine truncated
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAksAAAHHCAYAAACvJxw8AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAATk9JREFUeJzt3QecVOX1x/9DXaoUkd5ZOigGBEGNjYhCohijokaKCDZMBBVBRQRUFEuISiQmgpqoWKLGWBDsUfkJQVFQeu9Fel3Ynf/r+/z/d/6zs7uzhWl35vN+vcZlZu7cuXtn3Xv2POc5T6lAIBAwAAAA5Kt0/g8DAABACJYAAAAiIFgCAACIgGAJAAAgAoIlAACACAiWAAAAIiBYAgAAiIBgCQAAIAKCJQAAgAgIlgAkraZNm9rAgQPj8l56H70fik7n69e//nWiDwOIOYIlIM0sXLjQfve731mTJk2sQoUK1qBBA/vVr35lTz31VK7tHnroIXv77bcTdpwAkCwIloA08vXXX1uXLl3s+++/tyFDhtjTTz9t119/vZUuXdr+/Oc/59qWYAkA/l9l/7+vANLAgw8+aNWqVbN58+ZZ9erVcz23bds2S2WHDx+28uXLu8AwlRw8eNAqVaqU6MMAUlpq/dYAENHKlSutffv2eQIlqV27dvDfpUqVsgMHDtgLL7zg/q2bVzu0du1au/nmm61169ZWsWJFO/HEE+3yyy+3NWvW5Nrf888/71731Vdf2YgRI+ykk06yypUr26WXXmrbt2/PtW0gELAHHnjAGjZs6C785557rv344495jnHnzp12xx13WMeOHa1KlSp2wgkn2EUXXeQyZaE+++wz994zZsywe++91w01ar979+51zytj1qFDBzcMqa9vvfVWkc/hv//9b+vTp4/Vr1/fMjIyrEWLFjZhwgTLzs7Otd0555zj9j1//nzr0aOHO1fNmjWzqVOn5nusr776qt19991Wt25dd54uvvhiW79+fYH7/OUvf+m+J73GC3YHDx5sderUcd/XKaec4j6/cI899pg7Hn1uOqbOnTvbG2+8ke/3+s9//tO6du3q3qdGjRruPWfNmpVnuy+//NJtp/dt3ry5vfjii0U+n4AfkFkC0ojqlObMmWOLFi1yF92C/OMf/3DDc7oADh061D2moECUldJwXr9+/VxwoyDpmWeecRfyn376KU+W49Zbb3UX2rFjx7ptJ0+ebMOGDXPBgee+++5zwVLv3r3d7dtvv7ULLrjAsrKycu1r1apVLtBRcKbAY+vWrfbXv/7Vzj77bPfeCmBCKYhRNkkB1pEjR9y/dbG/7LLLrF27djZx4kT7+eefbdCgQe57KQoFgQrUFADq6yeffOKOX4HYo48+mmvbXbt2ue/niiuusKuuuspee+01u+mmm9xxXHfddXmyfgqa7rrrLhf46Dz17NnTFixY4IIaj45XAaLO/+9//3sXHB06dMid/xUrVrhzq3Pz+uuvuwB39+7d9sc//jH4eg23KhC75ppr3PlVQKnz+e6777og0DNu3Di7//77XWA1fvx4d8zffPON+3712Xj0nqqBU6A2YMAAmzZtmntfBWEKzIGUEACQNmbNmhUoU6aMu3Xv3j0wcuTIwIcffhjIysrKs23lypUDAwYMyPP4wYMH8zw2Z86cgH6dvPjii8HHpk+f7h7r2bNnICcnJ/j48OHD3fvv3r3b3d+2bVugfPnygT59+uTa7u6773avDz2Gw4cPB7Kzs3O99+rVqwMZGRmB8ePHBx/79NNP3WubN2+e53g7deoUqFevXvD9vfOi7Zs0aVLIGcz/+7/hhhsClSpVcsfnOfvss90+H3/88eBjR44cce9fu3bt4Dn3jrVBgwaBvXv3Brd97bXX3ON//vOf8+xz6tSpud5/8uTJ7vF//vOfwce0f33GVapUybXf8OPXdh06dAicd955wceWL18eKF26dODSSy/Nc75DPyOdL73vF198EXxMn6c+j9tvv72QMwn4B8NwQBrRrDdllpRZ0NDVpEmTrFevXm6Y6p133inSPkKzHEePHnWZjszMTDe0p4xQOGWmlDHxnHXWWW7ISsN58tFHH7kMhzJQodvddtttefalYS+v5kj70Hsru6MhwfzeW5mO0OPdvHmzy9TocdVuhZ4XZZqK+/3v27fPduzY4b4n1Q4tWbIk17Zly5a1G264IXhf2RndV+ZIQ2mh+vfvb1WrVg3eV7amXr169v777+c5B8qEhdI2Gr5T9spTrlw5+8Mf/mD79++3zz//PN/jV+Zrz5497vhDz5+ydzk5OS5jFl7jFfoZic6bXu/RcKs+D2UBgVRBsASkmdNOO83efPNNd6GcO3eujR492l30dXHWUFZhNOSji2ijRo3chbtWrVruAqnhHl14wzVu3DjXfQ3Jid5fvKCpZcuWubbTPr1tPbqA/+lPf3Lbhr73Dz/8kO97azgqVEHvJbrAF4VqqVR3pWBLNVN6fw2HSfgxaFhQ9UehWrVq5b6G13iFH5OCEgWh4dspsFXQFf596fXhgU3btm1zfd+i4bbTTz/d1RfVrFnTHb+GUUOPXbVt2ldRAsjwz1f0uXmfL5AKqFkC0pQuuAqcdNMFXNkK1bmotigSZYCmT5/uMj/du3d3QYMu7KqhUTATrkyZMvnuR0XdxaV2BmPGjHH1PqpH0sVeF3UdS37vHZpFiQYFhKqPUpCkOh7VcSnoUFZGtUb5HUO0Hc/39N///tdlFVWo/Ze//MVlrpSB0uf58ssvl2if0fx8gWRFsATA9V7yhqkKGm7xaOaUhrEef/zxXNPyFUiUtOhcli9f7mZSeTRjLjw7offWTLnnnnsu1+N6b2WZivNe4ZYuXVro6zVzTUN/yswp4PCsXr063+03bdrkZhWGZpeWLVvmvoZ3Cw8/JgUbKp4++eSTi/R9KbumYC00u+QNC3rf97/+9S8X3H344YcuM+dRsBRKQaD2pUxjp06dCn1/INUxDAekkU8//TTfv/i9upjQoShd4PMLgJRJCN+Hun+HT50vKs34UnZD+wjdr2aDFeW9lQ3buHFjkd5LmRRd/DWlPnTYafbs2UUagvSyKKHHoHorZWnyc+zYMTdbL3Rb3dfQl2aLhdJ0ew2HhgaGCl41860wmnG3ZcuWXDMM9d46p6rpUjbMO34FwaGflYb5wpuP9u3b1wVdyp6FZ8vIGCEdkVkC0oiG0FSIrJqbNm3auIu32gDoIqtMR2jhsC7mKr5+4oknXO2N6n+6devm1gJTawENv6mmRQXj2k59e0pCgYOm9msav/atC/93331nH3zwQZ5skZ7XBVzHqSntWrrlpZdeypWRKozeR1PkzzzzTDecp95NCio0zV3F0JHoPVWPo8yaiqcVeOhcFBRA6Lw98sgjLiDRUKfOswrMn332WRcghtKQoo5J35taIihYVM2SOq0XRkX0CsI0ZV+F4/osFWypx5X24xWO6/vW53nhhRfa1Vdf7QrNp0yZ4t5HmSmP7t9zzz1uqFPF27/97W9dJkptI/Q96RwCaSXR0/EAxM8HH3wQuO666wJt2rRxU8o1ZT8zMzNw6623BrZu3Zpr2yVLlgR++ctfBipWrJhrCv+uXbsCgwYNCtSqVcvto1evXm5bTSMPnebvtQ6YN29erv16U+X11aPp6ePGjXNT+vV+55xzTmDRokV59qmp+ZqS7m13xhlnuLYFmlKvW/h7vP766/meh3/961+Btm3buinu7dq1C7z55pvufYrSOuCrr74KnH766e7969evH2y/EP496Xjat28f+N///uem8FeoUMHt/+mnn873fLzyyiuB0aNHu7YC2rdaKaxduzbXtt4+86PPz/tc9Ll27NjRfQbhnnvuuUDLli3d966fA20zduxYdwzhpk2bFjj11FPdtjVq1HDvP3v27ODz+n50nOHCPw/A70rpP4kO2AAg1ahJpNoKqAFoYXVQqsPScKJmJAJIPtQsAQAARECwBAAAEAHBEgAAQATULAEAAERAZgkAACACgiUAAIAIaEoZBepwq2UN1PitoCUiAABAclElkjrnq9lq+ELUoQiWokCBklZgBwAA/rN+/Xpr2LBhgc8TLEWBt5SATrZWIwcAAMlv7969LtnhXccLQrAUBd7QmwIlgiUAAPylsBIaCrwBAAAiIFgCAACIgGAJAAAgAmqW4theICsrK9GHgTgoV66clSlTJtGHAQCIEoKlOFCQtHr1ahcwIT1Ur17d6tatS98tAEgBBEtxaHi1efNml2nQ9MRITa+QGp/3wYMHbdu2be5+vXr1En1IAIDjRLAUY8eOHXMXT3UHrVSpUqIPB3FQsWJF91UBU+3atRmSAwCfI80RY9nZ2e5r+fLlE30oiCMvMD569GiiDwUAcJwIluKE2pX0wucNAKmDYAkAACBVgqUvvvjCfvOb37j6H/3l/vbbbxf6ms8++8x+8YtfWEZGhmVmZtrzzz+fZ5spU6ZY06ZNrUKFCtatWzebO3dujL4DxJs+18mTJyf6MAAAPuarYOnAgQN2yimnuOCmKDRdv0+fPnbuuefaggUL7LbbbrPrr7/ePvzww+A2r776qo0YMcLGjh1r3377rdt/r169grOZ0tXAgQNdQKqb+gbVqVPHfvWrX9m0adOK1QJBwamm0SfKvHnzbOjQoUXeXsG1vufdu3fH9LgAAAXbuXOn/fjjj7Zo0aLgbfny5ZYovpoNd9FFF7lbUU2dOtWaNWtmjz/+uLvftm1b+/LLL+1Pf/qTC4jkiSeesCFDhtigQYOCr3nvvfdcUDBq1ChLZxdeeKFNnz7dFalv3brVZs6caX/84x/tjTfesHfeecfKlk3+H5+TTjop0YcAAIhg5cqVdujQISvMkSNHXNDUoUMHizdfZZaKa86cOdazZ89cjylI0uNes8j58+fn2kZ9kHTf26agD2zv3r25bvGgmVX79++P2wwrDV2qsWKDBg3cUObdd99t//7Line truncated
|
||||
"text/plain": [
|
||||
"<Figure size 432x288 with 1 Axes>"
|
||||
"<Figure size 640x480 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {
|
||||
"needs_background": "light"
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Results\n",
|
||||
"plt.plot(X,Y,'.',label='Data points', color=\"lightgray\")\n",
|
||||
"plt.plot(X,nn_sv.predict(X),'.',label='Supervised', color=\"red\")\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"plt.plot(X,Y,'.',label='Datapoints', color=\"lightgray\")\n",
|
||||
"plt.plot(X, nn_sup(torch.tensor(X)).detach(), '.',label='T', color=\"red\") \n",
|
||||
"plt.xlabel('x')\n",
|
||||
"plt.ylabel('y')\n",
|
||||
"plt.title('Standard approach')\n",
|
||||
@@ -230,7 +247,7 @@
|
||||
"Note that the red line is often not perfectly at zero, which is where the two modes of the solution should average out in the continuous setting. This is caused by the relatively coarse sampling with only 200 points in this example.\n",
|
||||
"<br>\n",
|
||||
"\n",
|
||||
"---"
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -254,7 +271,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"execution_count": 6,
|
||||
"id": "extensive-forward",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
@@ -264,13 +281,11 @@
|
||||
"\n",
|
||||
"# P maps Y back to X, simply by computing a square, as y is a TF tensor input, the square operation **2 will be differentiable\n",
|
||||
"def P(y):\n",
|
||||
" return y**2\n",
|
||||
" return torch.square(y)\n",
|
||||
"\n",
|
||||
"# Model\n",
|
||||
"nn_dp = tf.keras.models.Sequential([\n",
|
||||
" tf.keras.layers.Dense(10, activation=act, input_shape=(1,)),\n",
|
||||
" tf.keras.layers.Dense(10, activation=act),\n",
|
||||
" tf.keras.layers.Dense(1, activation='linear')])"
|
||||
"# Define custom loss function using the \"physics\" operator P\n",
|
||||
"def loss_function(y_true, y_pred):\n",
|
||||
" return criterion(y_true, P(y_pred))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -280,51 +295,53 @@
|
||||
"source": [
|
||||
"The loss function is the crucial point for training: we directly incorporate the function to learn, $f$ called `nn_dp`, into the loss. Keras will evaluate `nn_dp` for an inpupt from `X`, and provide the output in the second argument `y_from_nn_dp`. On this output, we'll run our \"solver\" `P`, and the result should match the correct answer `y_true`. In this simple case, the `loss_dp` function simply computes the square of the prediction `y_pred`. \n",
|
||||
"\n",
|
||||
"Later on, a lot more could happen here: we could evaluate finite-difference stencils on the predicted solution, or compute a whole implicit time-integration step of a solver. Here we have a simple _mean-squared error_ term of the form $|\\mathcal P(y_{\\text{pred}}) - x_{\\text{true}}|^2$, which we are minimizing during training. It's not necessary to make it so simple: the more knowledge and numerical methods we can incorporate, the better we can guide the training process."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "western-leader",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"#Loss\n",
|
||||
"mse = tf.keras.losses.MeanSquaredError()\n",
|
||||
"def loss_dp(x_true, y_from_nn_dp):\n",
|
||||
" return mse(x_true,P(y_from_nn_dp))\n",
|
||||
"Later on, a lot more could happen here: we could evaluate finite-difference stencils on the predicted solution, or compute a whole implicit time-integration step of a solver. Here we have a simple _mean-squared error_ term of the form $|\\mathcal P(y_{\\text{pred}}) - x_{\\text{true}}|^2$, which we are minimizing during training. It's not necessary to make it so simple: the more knowledge and numerical methods we can incorporate, the better we can guide the training process.\n",
|
||||
"\n",
|
||||
"optimizer_dp = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
|
||||
"nn_dp.compile(optimizer=optimizer_dp, loss=loss_dp)"
|
||||
"Let's instantiate the neural network again, and train the network with the _differentiable physics_ loss:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "artistic-table",
|
||||
"execution_count": 7,
|
||||
"id": "western-leader",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 1/5\n",
|
||||
"40/40 [==============================] - 0s 656us/step - loss: 0.2814\n",
|
||||
"Epoch 2/5\n",
|
||||
"40/40 [==============================] - 0s 1ms/step - loss: 0.1259\n",
|
||||
"Epoch 3/5\n",
|
||||
"40/40 [==============================] - 0s 962us/step - loss: 0.0038\n",
|
||||
"Epoch 4/5\n",
|
||||
"40/40 [==============================] - 0s 949us/step - loss: 0.0014\n",
|
||||
"Epoch 5/5\n",
|
||||
"40/40 [==============================] - 0s 645us/step - loss: 0.0012\n"
|
||||
"Epoch 10/50, Loss: 0.000004\n",
|
||||
"Epoch 20/50, Loss: 0.000003\n",
|
||||
"Epoch 30/50, Loss: 0.000003\n",
|
||||
"Epoch 40/50, Loss: 0.000002\n",
|
||||
"Epoch 50/50, Loss: 0.000002\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"#Training\n",
|
||||
"results_dp = nn_dp.fit(X, X, epochs=5, batch_size=5, verbose=1)"
|
||||
"nn_dp = SimpleNN(hiddendim=128)\n",
|
||||
"optimizer = optim.Adam(nn_dp.parameters(), lr=0.001)\n",
|
||||
"\n",
|
||||
"# Training loop\n",
|
||||
"batch_size = 5\n",
|
||||
"\n",
|
||||
"for epoch in range(epochs):\n",
|
||||
" permutation = torch.randperm(N)\n",
|
||||
" epoch_loss = 0.0\n",
|
||||
" \n",
|
||||
" for i in range(0, N, batch_size):\n",
|
||||
" indices = permutation[i:i+batch_size]\n",
|
||||
" batch_x = X_tensor[indices]\n",
|
||||
" \n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" outputs = nn_dp(batch_x)\n",
|
||||
" loss = loss_function(batch_x, outputs)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" \n",
|
||||
" epoch_loss += loss.item()\n",
|
||||
" \n",
|
||||
" if(epoch%10==9): print(f\"Epoch {epoch+1}/{epochs}, Loss: {epoch_loss/N:.6f}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -337,28 +354,25 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"execution_count": 8,
|
||||
"id": "indonesian-abraham",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEWCAYAAABIVsEJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAA8LUlEQVR4nO3deXxU9b34/9c7E0IIJCaBsIQtpEUKLgSN1LiGwlUuVcGqrVsRS0utRS3Wq7b2fmt35FZpb/V3FZcK9dalrljlqqApLlEJiqK4gCHsS0gCAYYQMnn//jhnxjOTmWwkmSzvp488mHPO55zzOZM47/nsoqoYY4wxLZUQ7wwYY4zpmiyAGGOMaRULIMYYY1rFAogxxphWsQBijDGmVSyAGGOMaRULIKbFROReEflPz/aPRGSXiBwQkf4icrqIrHe3Z8Qxq8H8fSwihc1MqyLy1RjHZonIG22ZN/e6Oe59E9v4ugdEJLctr9mVtdf73JPZG2nCiEgZMAioAwLAOmAJsEhV6wFU9RpP+l7AXcCpqvqBu+/XwN2q+ueOzT2IyMPAVlX9RXCfqh7X0fnoDFS1X7zzYLo3K4GYaM5X1VRgJDAfuAV4MEbaQUAy8LFn38iI7Wazb4fG/ga6DgsgJiZV3aeqS4HvAFeJyPHgfMsXkd+KyLHAZ27yvSLyqoh8AeQCz7tVKL1F5BgReVBEdojINvdcn3utWSLypogsFJEK4Hb3nD+KyGa3auxeEenjpi8Uka0i8lMR2e1e82r32BzgCuBm997Pu/vLRGSK+3qiiBSLyF733LtFJCni0aeJSKmI7BGR/xKRqP+fiMjXROQVEakUkc9E5Nux3ksRKRKRP4jIuyJSLSLPiUhmRLIr3GfeIyK3uecNFhG/iPT3XOskESkXkV4i8lUR+ZeI7HPPe9yTLlQdJyJ9ROROEdnkpn3D3ZcsIo+ISIX7nqwSkUExnuFWEflCRPaLyDoRudBzLPh7vNu9/qciMrk5z++pWpotIpuBV0UkQUR+4eZ3t4gsEZFjPNf7h4jsdO+1UkSO8xyL+qyNvc+mlVTVfuwn9AOUAVOi7N8M/Mh9/TDwW/d1DqBAYqxrAM8A9wF9gYHAu8AP3WOzcKrLrsOpUu0DLASWAplAKvA88Ac3faGb/tdAL2Aa4AcyIvMWLT/AycCp7r1ygE+An3jSKvCae+8RwOfA9z15fcN93RfYAlztXmsCsAcYF+N9LQK2Ace75z4FPBLxHt7vPv944DAw1j3+YvC9d7cXAn9xXz8K3IbzZTAZOCPiWb7qvr7HzcNQwAecBvQGfui+vynu/pOBtBjPcAmQ7d7rO8BBYEjE73Ge+3v5DrAPyGzB8y9xj/UBvgdswPky0g94GvibJy/fw/nb6A38CVjjORbrWRt9n+2nFZ8X8c6A/XSuH2IHkLeB29zXD9PMAIJTxXUY6OM5fhnwmvt6FrDZc0zcD6avePYVABvd14XAoYj77cZpgwnLW1PP5B77CfCMZ1uBqZ7ta4EVnrwGA8h3gNcjrnUf8MsY9ykC5nu2xwG17gdc8D0c5jn+LnCp515vuq99wE5goru9BFjkPTfiWb6K84F/CBgfJc33gLeAE1vxt7IGmO55b7YDEvEM323B8+d6jq8ArvVsjwGOeH/vnmPp7vnHNPGsjb7P9tPyH6trNM01FKhsxXkjcb6R7hCR4L4EnG/vQd7XWTjfhld70gvOB01QharWebb9ON9SmyROtdtdQL57n0RgdUQyb3424XzrjjQS+LqI7PXsSwT+1sjtI6/bCxjg2bfT89r7TM8B94rIKJwP0n2q+q577GbgN8C7IlIF3KmqD0XcdwBO6eSLKHn6GzAceExE0oFHcL4oHIlMKCIzgRtxPohx8+fN/zZ1P5U9z+h975p6fu/xbDeNN30iMEhEdgK/wykRZQH1nufs3cizBsV6n00LWRuIaZKInIITQFrThXULTglkgKqmuz9pGt4zyvuhswfnG+RxnvTHaPN7FDU1vfT/AJ8Co1U1Dfg5ToDyGu55PQLnm3WkLcC/PHlMV9V+qvqjRu4ded0jOM/bKFWtAZ4ArgS+iydIqepOVf2BqmbjVEf9f9KwG/IeoAb4SpRrH1HVX6nqOJyqnvOAmZHpRGQkTtXPXKC/qqYDHxH+3g0VT9Sn4XvX1PN7f3fbcYK0N30dsAu4HJgOTMEpdeQEs9nYs5q2ZwHExCQiaSJyHvAYTn312pZeQ1V3AC8Dd7rXSxCRr4jI2THS1+N8UC0UkYFuPoaKyLnNvOUunHrzWFKBauCAiHwNiPaB/x8ikiEiw4EbgMejpPkncKyIfNdtzO4lIqeIyNhG7n2liIwTkRScNpwnVTXQrKdyqqpmARfgCSAicomIDHM3q3A+hOu9J7rv6UPAXSKSLSI+ESkQp7PCJBE5QZxODdU4H+ph57v6utcud+97NU57htdA4Hr3vbgEGIvTftOa538UmCcio0SkH/B74HG35JmK86WkAqcU+fvmPGuM+5ijYAHERPO8iOzH+ZZ9G06Vz9VHcb2ZQBLOmJIq4ElgSCPpb8FpQH1bRKqB5ThVN83xIDDO7VH0bJTjN+F8g92PE6iiBYfncKq11gAvEKULs6ruB84BLsX5trwTuAOnCiWWv+G00ezEqWa5vhnPE7zfmzgf7O+pqrdq5xTgHRE5gNPx4AZVLY1yiZuAtcAqnKrIO3D+/x+M8/uoxulQ8C+iVMOp6jrgTqAYJ0ifALwZkewdYDROKeB3wMWqWuE53pLnf8hNvxLYiFOquM49tgSnSmsbzt/U2818VtPGJLzK0hjTHkSkCKcU98BRXONV4O9Hc432IiKzcHqrnRHjeBFH+fym87FGdGO6ALcd6iScun9jOgUr1hnTyYnIYpxqvJ+4VWfGdApWhWWMMaZVrARijDGmVXpUG8iAAQM0Jycn3tkwxpguZfXq1XtUNStyf48KIDk5OZSUlMQ7G8YY06WIyKZo+60KyxhjTKtYADHGGNMqFkCMMca0So9qA4nmyJEjbN26lZqamnhnpdtKTk5m2LBh9OrVK95ZMca0obgGEBF5CGf2z92qGjkxG+7Mnn/my0WDZqnqe+6xq4Dgute/VdXFrcnD1q1bSU1NJScnh/CJRE1bUFUqKirYunUro0aNind2jDFtKN5VWA8DUxs5/u84k7ONBubgTMWNuxTmL4GvAxOBX4pIRmsyUFNTQ//+/S14tBMRoX///lbCM6YbimsAUdWVNL5I0XRgiTreBtJFZAhwLvCKqlaqahXwCo0HokZZ8Ghf9v4a0/H8fj/l5eX4/f52u0dnbwMZSvgqZVvdfbH2NyAic3BKL4wYMaJ9cmmMMZ2E3++nqqqKvXv3oqqICKNGjSIlJaXN7xXvKqx2p6qLVDVfVfOzshoMpOwUfD4feXl5HHfccYwfP54777yT+vpoa/p8qaysjL///e/tkp9p06axd+/eRtM8/PDDbN8ebaE+Y0xb8vv9bNu2jW3btrFz507KysqorIxeceP3+9m4cSNVVVXBNd9RVQ4ePNgueevsJZBthC+DOczdtw0ojNhf1GG5amN9+vRhzZo1AOzevZvLL7+c6upqfvWrX8U8JxhALr/88jbPz4svvthkmocffpjjjz+e7Oxoy4UbY1rK7/dz8OBBfD4fgUAAn8/HoUOHqKqqapD2wIEDAGRmZobtP3jwIJET5IoIffv2bZc8d/YSyFJgpjhOBfa5S6S+BJzjLjuagbMy3Esdlan2rFscOHAgixYt4u6770ZVKSsr48wzz+Skk07ipJNO4q233gLg1ltv5fXXXycvL4+FCxfGTFdUVMRZZ53FN7/5TcaMGcM111wTKt08+uijnHDCCRx//PHccsstoTzk5OSwZ88eysrKGDt2LD/4wQ847rjjOOecczh06BBPPvkkJSUlXHHFFeTl5XHo0CFuvfVWxo0bx4knnshNN93U5u+LMd2N93MkWHLYtWsX27dvD/0bLXgEVVdXN9jXt2/fsDbHjIyMdqu+gvh3430UpyQxQES24vSs6gWgqvfirKc8DWd5Uz/usqqqWikiv8FZshLg16raWGN8mwn+otuzbjE3N5dAIMDu3bsZOHAgr7zyCsnJyaxfv57LLruMkpIS5s+fzx//+Ef++c9/hvIVLR3Au+++y7p16xg5ciRTp07l6aef5rTTTuOWW25h9erVZGRkcM455/Dss88yY8aMsLysX7+eRx99lPvvv59vf/vbPPXUU1x55ZXcfffd/PGPfyQ/P5+KigqeeeYZPv30U0SkyeovY3qaYOkiWBKoqqoKBQcRIT09vUHJoSlpaWkN9qWkpDBq1KjQvdorcATFNYCo6mVNHFfgxzGOPYSzbnKH8hYRg3WL7flLOnLkCHPnzmXNmjX4fD4+//zzFqebOHEiubm5AFx22WW88cYb9OrVi8LCQoLtQldccQUrV65sEEBGjRpFXl4eACeffDJlZWUN7n3MMceQnJzM7NmzOe+88zjvvPOO/sGN6WIqKyuprq4mLS0trGrJ+6UTnIDhDRax9geJCP369SMxMRGfz0dNTU2De3ilpKS0e+AI6uxtIJ1OsIgYLIG0R91iaWkpPp+PgQMH8qtf/YpBgwbxwQcfUF9fT3JyctRzFi5cGDNdZDfalnSr7d27d+h1sE42UmJiIu+++y4rVqzgySef5O677+bVV19t9j2M6eoqKytDnUoi2yci2yWitVFkZGSQkZHRoA0kEAh0SEmitSyAtFB7FxHLy8u55pprmDt3LiLCvn37GDZsGAkJCSxevJhAIABAamoq+/d/ubpprHTgVGFt3LiRkSNH8vjjjzNnzhwmTpzI9ddfz549e8jIyODRRx/luuuua3Y+vfc/cOAAfr+fadOmcfrpp4dKO8Z0Vd4qp5qamqglC6/I9ojq6upQWu+XTiDsC2h6ejoZGRmhz5HOGihisQDSCm1dRDx06BB5eXkcOXKExMREvvvLine truncated
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAksAAAHHCAYAAACvJxw8AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAVhxJREFUeJzt3Qd4VFX6x/E3PfQqvUR6FwVpdmEB0RVYdEFQyiKsBf8KKE0BAQVBRURRRETQRUFRUVFRQJFVWUAURZr03ntNgMz/ec8+d3YymUwmybQ78/08zxhm5s6dkzsx95dz3nNujMPhcAgAAAA8ivX8MAAAABRhCQAAwAvCEgAAgBeEJQAAAC8ISwAAAF4QlgAAALwgLAEAAHhBWAIAAPCCsAQAAOAFYQnwo6efflpiYmIyPHbp0iUZNGiQVKxYUWJjY6VDhw7m8TNnzsj9998vZcqUMa957LHHxG527Nhh2j5z5sxcH6sjR45ku21KSor07NlTgk2/L23jzz//bItjirzJyc8kokt8qBsAhCs9WfXq1ct5PykpSYoXLy7169eX22+/3TxXqFChbPczY8YMef75500Yuuaaa6RSpUrm8bFjx5r3GD58uFStWlVq164t4eq9996TQ4cO2TLQAUBeEZaAbIwePVquvPJKuXjxohw4cECWLl1qQsPEiRPls88+kwYNGji3feqpp2TIkCEZXv/tt99K+fLl5aWXXsr0eLNmzWTkyJES7jQs/fHHH5nCUuXKleX8+fOSkJAQsrZFGo4pEH4IS0A2brvtNmncuLHz/tChQ03QueOOO+TOO++UDRs2SL58+cxz8fHx5uZKe2SKFi2aab/6eJ06dfzWzvT0dElLS5Pk5GQJFh2yCOb7RYNoP6Znz56VAgUKhLoZQAbULAG5cOutt5rhs507d8q//vUvjzVLVu3Jd999J+vWrTP/1pv2TOnX7du3yxdffOF8XLdXqampprepWrVqZuhPa5205kkfd6Wv6devn8yePVvq1q1rtl24cKF5bu/evfKPf/xDSpcubR7X53U40JXVjg8++ECeffZZqVChgjlJt2zZUrZs2eLc7uabbzbt1O/VaqvWEGVVX/P777+b+qIqVaqY/WlNlrbl6NGjHo+l1of8/e9/l8KFC0uJEiXk0UcflQsXLmT7GZw4ccL0dOnx0e9Rj9f48eNNaMyOtl/D7jfffCMNGzY07dTg+vHHH3vcXo/9gAED5IorrjAn8o4dO8rhw4edz/fo0UNKlixpeh/dtW7dWmrWrOm8v2jRIrn++utNgC5YsKB5btiwYdnWLG3cuNEcJ22DhnN93ZNPPul8/vTp0+Z46Pemx6NUqVLyl7/8RX755Revx0I/14ceesjsT/ern8Hdd9/t/Hl0r99atmyZ/POf/zTb6WfWvXt3OX78eK6Or7XP77//3rRB26w/h5bXXnvN+bNdrlw5efjhh83n7urf//63aa8Ob1v/v/Tv39/0zrnL7hha9D30Z1g/oyJFipgh93Pnznk9johs9CwBuXTfffeZk5yeEPr06ZPpef2F/O6775ogosXc48aNM49rbZI+rr/Q9cQwcOBA5/Z6otfeqh9++EH69u1rtl27dq0Zwvvzzz9l/vz5Gd5De7g07Gho0pO1nqQOHjxohvesMKX7/eqrr6R3795y6tSpTENpzz33nCk8f/zxx+XkyZMyYcIE6datm6xYscI8rycTfXzPnj3OoUQ9yWdFw8C2bdvMCUaDkgbFadOmma//+c9/MhXA68lL263HR5+fPHmyOfm+8847Wb6HnrhuuukmEwr1xK0nyp9++sn0+u3fv18mTZqU7ee3efNm6dy5szzwwAMm7Lz99tvmpKuBU0OGq0ceeUSKFStmQqyGCN2/Htu5c+c6fxa0vV9//bUJCRYdttXPyBpq1WOgz+vQrQ7v6sldg+mPP/7ota0aQG+44QYzNKc/F3q8tm7dKp9//rn5+VL6fcybN8+0S4OJhlP9OdKeT62Vy8qqVavMsevSpYv5edTv7/XXXzchef369ZI/f/4M2+v+NUToHwabNm0y22rgssJ3bo6vBiX9OR0xYoTpWVK6/1GjRkmrVq3kwQcfdL6XtlePlzVM+eGHH5qfB91GA9zKlSvllVdeMT+v+lxOjqHrz6QOvevPpIbN6dOnmyCnYRxRygHAo7ffftuh/4usWrUqy22KFCniuPrqq533R44caV7j6qabbnLUrVs302srV67suP322zM89u677zpiY2Md//73vzM8PnXqVLPfH3/80fmY3tdt161bl2Hb3r17O8qWLes4cuRIhse7dOli2nvu3Dlz/7vvvjP7qF27tiM1NdW53csvv2weX7t2rfMxbae219327dvNtnqsLNb+Xb3//vtmu2XLlmU6VnfeeWeGbR966CHz+G+//ZbhWPXo0cN5f8yYMY4CBQo4/vzzzwyvHTJkiCMuLs6xa9cuhze6P32Pjz76yPnYyZMnzXFz/Tytn4FWrVo50tPTnY/379/fvM+JEyfM/cuXLzsqVKjg6Ny5c4b3mThxoiMmJsaxbds2c/+ll14y+zt8+HCWbfN0TG+88UZHoUKFHDt37sywrWub9LN9+OGHHTnl6fNavny5acM777yT6Vg0atTIkZaW5nx8woQJ5vFPP/0018f3+uuvd1y6dMn5+KFDhxyJiYmO1q1bm2NrefXVV832M2bM8Nr+cePGmePuerx8OYbWz+Q//vGPDNt07NjRUaJEiSyOIKIBw3BAHmgPiw5/+Iv+Jay9SbVq1TLDU9ZNh/2UDum50t4V17onzVAfffSR/PWvfzX/dt1HmzZtTA+R+7CM9gAlJiY67+tf30p7h3LDqt9SOpym7609XcrTkJAOrbj34qgvv/zS63HSdmpvj+v3qL0Qly9fNkNF2dFhHR1Os1hDSr/++qvpEXKlPRGuvSb63vo+2qOitGdOe+O04N/150GHSFu0aGF6KZRVu/bpp5/6NFyodLhPvx8dyrRmUlpc26T71t7Affv2SW4/Lx1G1B4pHdLU/Xn6vPRYuBafa4+O1um5f145Ob7aMxsXF+e8v3jxYlN/p72gemxdt9P96LCwp/Zrr5T+HOgx159/fa+cHEOL9oa50s9bj4v2zCI6EZaAPNDhNV+WD/CVDl3oUI0OSbjeatSo4SwKd2WdhC16UtB6Cx32ct+HtQyC+z7cTx4aQJR7HYqvjh07ZuqOtF5KT2T63lY7Nay5q169eob7uoyCniDda2bcj5MO57h/jxqWPH2PnmggcD9RWsfZ/b19OUYaBLRO5pNPPjH3ddho9erVZojOosNS1113nVlfS4+PDn3pMKq34GSF1nr16nn9fnT4VGcsas1OkyZNzDCWL4FX26zDX1btlw7n6rHUnyNfPi/9g6Fs2bKZjllOjq/7z7EVQl1rvZSGeq2Fs55Xu3btMvVFuqyHtkXbrn9EKKv9vh7DQP0/AfujZgnIJa2J0F/GelLwFz1p6jpOuiyBJ3pCc+X6V7X1enXvvfeaOhFPXJc6UK5/0bv670hfzmm9h9bAPPHEE6a4V09g2q62bdv61Jvi6S99d7ofrXvRwndPrJOyv/hyjLSHr1GjRqbgX4OTftWTux4P189Lezi0h1B7RzTwad2T9hxq7VtW7+MLfR/tAdGwpvvStb20xkaLqnVGZ1a0J0/ribQXp3nz5qagWT8DDXK+9n7llfvPsa+0d09/DjSgDx482PTIagG+1rJpgMpt+/39/wTsj7AE5JIWaSsd3vIX7VX57bffzIw0X0KDO/2rWnu69CRi9bL4g69t0b+8lyxZYgpztbfCtScoK/qca8+CFjzrSc6acZfVcdJevbx8j/o+evJz/d60iF55e29vNCTprDktMte1qXTxUqtXwqK9Zvr56k1DsS5OqkX0GqA8fT/ak6K01yg72sOjxdJ60941LezW4mVvYUmLwjVYv/jiixmGT91nnbl+Xrfccovzvn4O+v22a9fOb8dX15qyeues71/p0JzOIrWOk05+0H3OmjXLHHvXSQa5PYaAJwzDAbmgM5zGjBljTvJaq+Iv2jugfxW/+eabHodLrJlC3v4i7tSpk6lb8nRicJ3unhP617qnIRlP7+/pL3Bvs9OmTJmS4b7OZFLeTvB6nJYvX25mn7nTk7xeYiY7WttjDZkprUfRGW3aG6az+HLjnnvuMeFAhyF16Ed7+FxpD4g7fT/lvjSEawC+8cYbzdIPOuTkyjrOGo7dPx+dvaV1Q1nt1/Uzc/+89DPQfXqiQ7yuSyToDDU93u6fV16Or4Yh7ZXTmZGubXvrrbfM96kh1Gq763Gw/v3yyy/n+BgC3tCzBGRDp93r+ix6QtBp+RqU9C9X/etXC3r9uYCg1rdoDYsWmGpPg9a36ElL318f13DgukCmJ7oUgL62adOmpiBWh4f0JK3Fulo46+mEnR0dXtLhIu01ufbaa83QmhaRu9PiWz0paf2MnlB15XIdEtLegKzoc7pcgg7TaQDS4auuXbvKVVddleVrdIhPj71Ow9fhFm2fBkntadCeEq2J0dobb3SoTpdT0KnoWj+kJ1L9fHVIKrf0pKzfhxaga4G0dVK36HIBOgynj+vPj/b+6FpCOmVf117KioYGfV57irTAWkO6fo86lLdmzRpTVK77uOuuu8xx089HP2v93lx7jDzRY6i9pDr8pj8r+hnoa3Uavifau6O9YhpYtedH269t08/QX8dXj6MuA6E9lHo8dd/We+nPnxVCddhNexl12Qv9I0N//vQPBU+1RdkdQ8CrUE/HA8KVNa3ZuulU5jJlyjj+8pe/mOn1p06dyvSavC4doHRa9vjx481rkpKSHMWKFTPTtUeNGmWmX1v0fbKaKn7w4EHzXMWKFR0JCQmm3S1btnRMmzbNuY21dMCHH36Y7dT1M2fOOLp27eooWrSoec5aRsDTtnv27DFTrXVbnc5+9913O/bt22e20+PjfqzWr1/Line truncated
|
||||
"text/plain": [
|
||||
"<Figure size 432x288 with 1 Axes>"
|
||||
"<Figure size 640x480 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {
|
||||
"needs_background": "light"
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Results\n",
|
||||
"plt.plot(X,Y,'.',label='Datapoints', color=\"lightgray\")\n",
|
||||
"#plt.plot(X,nn_sv.predict(X),'.',label='Supervised', color=\"red\") # optional for comparison\n",
|
||||
"plt.plot(X,nn_dp.predict(X),'.',label='Diff. Phys.', color=\"green\") \n",
|
||||
"plt.plot(X, nn_dp(torch.tensor(X)).detach(), '.',label='T', color=\"green\") \n",
|
||||
"plt.xlabel('x')\n",
|
||||
"plt.ylabel('y')\n",
|
||||
"plt.title('Differentiable physics approach')\n",
|
||||
@@ -371,15 +385,13 @@
|
||||
"id": "star-radio",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This looks much better 😎, at least in the range of 0.1 to 1. \n",
|
||||
"This looks much better 😎, at least if we're avoiding the origin (this part would need some extra attention).\n",
|
||||
"\n",
|
||||
"What has happened here?\n",
|
||||
"\n",
|
||||
"- We've prevented an undesired averaging of multiple modes in the solution by evaluating our discrete model w.r.t. current prediction of the network, rather than using a pre-computed solution. This lets us find the best mode near the network prediction, and prevents an averaging of the modes that exist in the solution manifold.\n",
|
||||
"\n",
|
||||
"- We're still only getting one side of the curve! This is to be expected because we're representing the solutions with a deterministic function. Hence, we can only represent a single mode. Interestingly, whether it's the top or bottom mode is determined by the random initialization of the weights in $f$ - run the example a couple of times to see this effect in action. To capture multiple modes we'd need to extend the NN to capture the full distribution of the outputs and parametrize it with additional dimensions.\n",
|
||||
"\n",
|
||||
"- The region with $x$ near zero is typically still off in this example. The network essentially learns a linear approximation of one half of the parabola here. This is partially caused by the weak neural network: it is very small and shallow. In addition, the evenly spread of sample points along the x-axis bias the NN towards the larger $y$ values. These contribute more to the loss, and hence the network invests most of its resources to reduce the error in this region.\n"
|
||||
"- We're still only getting one side of the curve! This is to be expected because we're representing the solutions with a deterministic function. Hence, we can only represent a single mode. Interestingly, whether it's the top or bottom mode is determined by the random initialization of the weights in $f$ - run the example a couple of times to see this effect in action. To capture multiple modes we'd need to extend the NN to capture the full distribution of the outputs and parametrize it with additional dimensions.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -387,6 +399,289 @@
|
||||
"id": "necessary-filename",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# A Probabilistic Generative AI Approach\n",
|
||||
"\n",
|
||||
"As hinted at above, we can do even better with state of the art AI techniques: we can learn the full _distribution_ of the posterior, in our case the different answers for each $x$. Below, we'll use _flow matching_ as a state of the art approach from generative, diffusion-based algorithms.\n",
|
||||
"\n",
|
||||
"As these methods work with noisy data, we first need to specify a new dataloader, that adds different amounts of \"noise\" onto the y values of our data, so that the network can learn the right direction towards the two possible modes."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "80a502f5-7b36-4a83-9a02-023d3d229a99",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from torch.utils.data import Dataset, DataLoader\n",
|
||||
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
|
||||
"\n",
|
||||
"class FlowMatchingDataset(Dataset):\n",
|
||||
" def __init__(self, data_x, data_y, n_samples=1000, sigma_min=1e-4):\n",
|
||||
" super().__init__()\n",
|
||||
" self.n_samples = n_samples\n",
|
||||
" self.sigma_min = sigma_min\n",
|
||||
" self.data_x = data_x\n",
|
||||
" self.data_y = data_y \n",
|
||||
"\n",
|
||||
" def __len__(self):\n",
|
||||
" return self.n_samples\n",
|
||||
"\n",
|
||||
" def __getitem__(self, idx):\n",
|
||||
" x0 = np.random.multivariate_normal([0.0, 0.0], np.eye(2), 1)[0]\n",
|
||||
" t = np.random.rand() # scalar in [0,1]\n",
|
||||
" dx = self.data_x[idx] #:idx+1]\n",
|
||||
" dy_org = self.data_y[idx]# :idx+1]\n",
|
||||
" x0[0] = dx[0] # keep x value\n",
|
||||
" x1 = np.concatenate([dx,dy_org],axis=0)\n",
|
||||
" #print([self.data_x.shape,dx.shape,x1.shape])\n",
|
||||
"\n",
|
||||
" x_t = (1 - ( 1 - self.sigma_min) * t) * x0 + t * x1\n",
|
||||
" u_t = (x1 - x0)\n",
|
||||
" x_t = torch.tensor(x_t, dtype=torch.float32)\n",
|
||||
" t = torch.tensor([t], dtype=torch.float32)\n",
|
||||
" u_t = torch.tensor(u_t, dtype=torch.float32)\n",
|
||||
" return x_t, t, u_t"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "12554942-2e9f-4d7d-b8e5-bff4ef95a8fe",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The network itself is not much different from before, we only need to add an additional time input `t`:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "e025dc6d-2ca2-4709-b1d4-baf76302dca5",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class VelocityNet(nn.Module):\n",
|
||||
" def __init__(self, hidden_dim, in_dim=2, time_dim=1, out_dim=2):\n",
|
||||
" super().__init__()\n",
|
||||
" self.net = nn.Sequential(\n",
|
||||
" nn.Linear(in_dim + time_dim, hidden_dim),\n",
|
||||
" nn.ReLU(),\n",
|
||||
" nn.Linear(hidden_dim, hidden_dim),\n",
|
||||
" nn.ReLU(),\n",
|
||||
" nn.Linear(hidden_dim, out_dim)\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" def forward(self, x, t):\n",
|
||||
" xt = torch.cat([x, t], dim=1)\n",
|
||||
" return self.net(xt)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9ba9acfa-5a58-4414-8873-0b07129dd217",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Training proceeds in line with before, we simply sample noisy samples from the dataset, and train the network to move samples towards the solutions in the dataset:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "cd463858-d0aa-49df-90c3-1be4daa46f30",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 10/50, Loss: 0.3837\n",
|
||||
"Epoch 20/50, Loss: 0.3773\n",
|
||||
"Epoch 30/50, Loss: 0.3735\n",
|
||||
"Epoch 40/50, Loss: 0.3726\n",
|
||||
"Epoch 50/50, Loss: 0.3715\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"batch_size = 128\n",
|
||||
"\n",
|
||||
"dataset = FlowMatchingDataset(X, Y, n_samples=N)\n",
|
||||
"dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)\n",
|
||||
"\n",
|
||||
"nn_fm = VelocityNet(hidden_dim=128).to(device)\n",
|
||||
"optimizer = optim.Adam(nn_fm.parameters(), lr=0.001)\n",
|
||||
"criterion = nn.MSELoss()\n",
|
||||
"\n",
|
||||
"for epoch in range(epochs):\n",
|
||||
" running_loss = 0.0\n",
|
||||
" for x_t, t, u_t in dataloader:\n",
|
||||
" x_t = x_t.to(device)\n",
|
||||
" t = t.to(device)\n",
|
||||
" u_t = u_t.to(device)\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" pred_v = nn_fm(x_t, t)\n",
|
||||
" loss = criterion(pred_v, u_t)\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
"\n",
|
||||
" running_loss += loss.item() * x_t.size(0)\n",
|
||||
" running_loss /= len(dataset)\n",
|
||||
" if(epoch%10==9): print(f\"Epoch {epoch + 1}/{epochs}, Loss: {running_loss:.4f}\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1a7ca01b-b501-4273-930c-2c4bb3d98c41",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"For evaluation, we now repeatedly call the neural network to improve an initial noisy sample drawn from a simple distribution, and step by step move it towards a \"correct\" solution. This is done in the `integrate_flow` function below."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "1837e971-7dd2-4e74-a9bb-82a21dc04195",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\n",
|
||||
"def integrate_flow(nn, x0, t_span=(0.0, 1.0), n_steps=100):\n",
|
||||
" trajectory = []\n",
|
||||
" t = torch.linspace(t_span[0], t_span[1], n_steps).to(x0.device)\n",
|
||||
" dt = 1./n_steps\n",
|
||||
" x_in = x0\n",
|
||||
" for i in range(n_steps):\n",
|
||||
" x0 = x0 + dt * nn(x0, torch.tensor([i/n_steps]).expand(x0.shape[0], 1) )\n",
|
||||
" x0[:,0] = x_in[:,0] # condition on original x position\n",
|
||||
" trajectory.append(x0)\n",
|
||||
" return trajectory, t\n",
|
||||
"\n",
|
||||
"# Generate samples along x, then randomize along y\n",
|
||||
"n_gen = 500\n",
|
||||
"x_in = torch.linspace(0.,1., n_gen).to(device)\n",
|
||||
"y_in = torch.randn(n_gen).to(device) * 0.95\n",
|
||||
"x0_gen = torch.stack([x_in,y_in],axis=-1)\n",
|
||||
"trajectory, time_points = integrate_flow(nn_fm, x0_gen)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "90ed7951-645b-45ee-b1ab-adada71382d5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"To illustrate this flow process, the next cell shows samples at different times in the flow integration. The initial random distribution slowlyl transforms into the bi-modal one for our parabola targets."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"id": "8d8f7d39-7df1-48ef-9ae6-8542c9a692e7",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABcsAAAG7CAYAAAAc+c0EAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQABAABJREFUeJzs3Qd8nFeVN/6fpvc+o1Hv3XKviWM7dhzSCEkIAQIhoQVYdmFhl/4u7d19gf+yu1kWyIYaSCFAKEkgIcWp7r2oy+oaSdN7b//PvYqdOHGRZI00ks6Xz4My4ymPHo/nPs+5555TkM1msyCEEEIIIYQQQgghhBBCljDBfO8AIYQQQgghhBBCCCGEEDLfKFhOCCGEEEIIIYQQQgghZMmjYDkhhBBCCCGEEEIIIYSQJY+C5YQQQgghhBBCCCGEEEKWPAqWE0IIIYQQQgghhBBCCFnyKFhOCCGEEEIIIYQQQgghZMmjYDkhhBBCCCGEEEIIIYSQJY+C5YQQQgghhBBCCCGEEEKWPAqWE5LnstnsfO8CIYQQQmaIxnFCCCFkYaOxnJClhYLlhOSxXbt24Utf+lLOXj8cDuNb3/oWrrzySqxatQof//jH0d/ff8nntbe388du3LgRGzZswEc+8hF+35ulUincd9992Lp1K1asWIE777wTJ06cyNnvQgghhOSbfB3HA4EAvvnNb5593nvf+17s27fvnMfQOE4IIYTk71je3d2Nj33sY1i/fj02b97M99Hlcp3zGBrLCZkZCpYTkscefPBBjI+P5+z1/+mf/gl/+9vf+M/vfe97sNvt+NCHPgS/33/B5wwNDeGDH/wgYrEY/u3f/g3f+c53kEgk+MD75kH9u9/9Lt9/NoD/13/9F4RCIe655x7+fEIIIWQpyMdxPJ1O8wtxdvH/hS98AT/4wQ+g0Whw7733oqur6+zjaBwnhBBC8nMsZ0Hxu+++G263m1+Pf/WrX8WhQ4f4+J5MJs8+jsZyQmaGguWELFHHjh3DSy+9xAfQ2267Dddeey0fSNnM9qOPPnrB5z300EOQy+V44IEHsGPHDmzfvh0/+clP+H0PP/wwfww7mfjNb36DL37xi7jrrrv4Y37+859Dp9Phpz/96Rz+loQQQsjiNNNx/KmnnkJbWxt+9rOf4ZZbbsFVV12FH/7whyguLsaePXv4Y2gcJ4QQQvJ3LGcT3l6vF//93//Nr8lvuOEGnp3e0dHBX5OhsZyQmRNdxnMJITnEBrSDBw/y/25oaMCvf/1rXvLkrdigZ7PZLro863x2794NhULBl2ydYTAYsG7dOrzyyiv41Kc+dd7nVVdX87Ir7LlnsP+2Wq0YHh7mt9lSbrbka+fOnWcfI5FIsG3bNrzwwgtT+v0JIYSQhSxfx/Fnn32WP4bt0xlSqZTffwaN44QQQkj+juXxeJz/VKlUZ+9jQXDG5/PxnzSWEzJzFCwnJE994xvf4Mujz/x3bW3teR/HssFYGZTp6uvrQ2lpKV+K9Wbl5eU86+xCWLmVt2LLuHp7e7Fp06azr61UKmE2m895XEVFBRwOB58pZ39OCCGELFb5Oo6zUissC41lrrGLfrbcmwUA2BLutWvXnn1tGscJIYQsdfk6ll9//fV8dfe3v/1tPn6z4Pn/9//9f3zcvuKKK86+No3lhMwMBcsJyVNsID4zU7xy5coLPq65uXlGrx8MBs+ZiT6DDZhs4JwqVrucNRNhs9SslvmlXpsJhUI0MBNCCFnU8nUc93g8vDaqVqvlS7NZGTV2wc1Wjf3ud79DY2MjjeOEEEJIHo/lLADOyq58/vOfxzPPPMPvY+M6mwQ/83o0lhMycxQsJ2SBY426stnsBf9cJDr/P/OLPaegoGBK780G2E9/+tM4deoUr5dWUlJyyddmBAJql0AIIYTMxzjOGn+xC+jHH3+cl1Bj1qxZw5dpsxqm//Ef/0HjOCGEEJLHYznLOmcT3tdddx3e/e5388zyX/ziF3zim/UYq6mpobGckMtAwXJCFjh2cTuT+mhslpl10X4rNoOtVqsv+b6sYcgnPvEJDAwM8M7a11xzzTmvfb6ZcBZcZ6by+oQQQshSMNfjOMsiYxfRZwLlZ15r1apVvDHYmds0jhNCCCH5OZazsi9s3GbX4WdceeWVvNEnS2L7wQ9+QGM5IZeBguWELHD333//jOqjVVVV8YYimUzmnFllVn+cXURfDBvsP/rRj56dwWYNSN7aBJQNwmypN2tQ8ubXZtnnMpls2vtLCCGELEZzPY6zWqXnez/WBOzM+EzjOCGEEJK/YzkLzL85WY1hY/OyZct4LzGGxnJCZo7WXRCSx6ayNIo15Wptbb3gdiGs4zabaX7ttdfO3scG0sOHD/NZ6YtllH/4wx/my8J+85vfvC1QzpxpKsJqop7BTh5efvnli742IYQQspjk4zi+detWdHZ28sZfZ3i9Xhw9epSXY2FoHCeEEELydyxngXA2br+51ApLZGtvb0dZWRm/TWM5ITNHmeWE5DGNRoNjx45h3759vGkIa9oxW1iQe/369by7N9t0Oh3+53/+hy/Hev/733/2cadPn+aD6pmmJf/6r/8Kt9vNG4qwmerjx4+ffSxb6sWaoLCZ6ltvvRXf+c53+KBdWVmJX/7ylwgEAvjYxz42a78DIYQQks/ycRz/0Ic+hD/+8Y+499578bnPfY43+GQZcWwSnK0aY2gcJ4QQQvJ3LP/sZz/Le4exn7fffjv/s1/96lew2+289whDYzkhM1eQvVTVf0LIvNm/fz++8pWvwOl08kHune9856y+vt/vx3e/+1288MILfOnX6tWr+fuxmeoz7rrrLr7M68UXX+SDMKuNxpZqnw8b6FlDEYY99vvf/z7+8pe/IBKJoKWlhTchWbFixaz+DoQQQki+yrdx/IyJiQn8+7//O89kYw0/2fO+/OUvo66u7uxjaBwnhBBC8ncsf/XVV/HjH/+Y9xth/UiWL1/OJ8EbGxvPPobGckJmhoLlhBBCCCGEEEIIIYQQQpY8qllOCCGEEEIIIYQQQgghZMmjYDkhhBBCCCGEEEIIIYSQJY+C5YQQQgghhBBCCCGEEEKWPAqWE0IIIYQQQgghhBBCCFnyKFhOCCGEEEIIIYQQQgghZMkTIc+tXbsWiUQCZrN5vneFEEIIuSxOpxMSiQSHDx/GUkHjOCGEkMWCxnFCCCFk8Y/jeZ9ZHo/HkUqlZuW1stksfz32k0wfHb+Zo2N3eej4zRwdu/w6fmw8Y6+3lMzmOM7QZ3rm6NhdHjp+M0fHbubo2F0eGscvH43j+YOO3eWh4zdzdOwuDx2/hTeO531mucVi4T937dp12a8ViUTQ2dmJpqYmKBSKWdi7pYWO38zRsbs8dPxmjo5dfh2/HTt2YKmZzXGcoc/0zNGxuzx0/GaOjt3M0bG7PDSOXz4ax/MHHbvLQ8dv5ujYXR46fgtvHM/7zHJCCCGEEEIIIYQQQgghJNcoWE4IIYQQQgghhBBCCCFkyaNgOSGEEEIIIYQQQgghhJAlj4LlhBBCCCGEEEIIIYQQQpY8CpYTQgghS8QDDzyAu+6666KPefLJJ9HQ0PC2bXR0dM72kxBCCCGEEEIImQ+ieXlXQgghhMypRx55BPfddx/Wrl170cd1d3dj/fr1+M///M9z7jcYDDneQ0IIIYQQQgghZH5RsJwQQghZxOx2O77xjW/gwIEDqKysvOTje3p6eCa52Wyek/0jhBBCCCGEEELyBQXLCSGEkEWsvb0dYrGYl1f50Y9+BJvNdsnM8u3bt0/rPXbs2HHBPxsfH4fVakUkEsFsiEaj5/wkU0fH7vLQ8Zs5OnYzR8cuv45fNptFQUHBrLwWIYQQQvITBcsJIYSQRYwFvqca/Pb7/TwT/fDhw3j00Ufh9XqxfPlyfOELX0BVVdWM9yGRSKCzsxOzaXBwcFZfbymhY3d56PjNHB27maNjlx/Hj41nUql0Vl6LEEIIIfmJguWEEEII4Xp7e89mzn3nO99BLBbD/fffjzvvvBNPPfUUTCbTeZ+3a9eui2ads9dramqalX1k2YEs6MFKysjl8ll5zaWCjt3loeM3c3TsZo6OXX4dP4lEMiv7RQghhJD8RcFyQgghhHCs+ee+ffug1+vPLjP/4Q9/iG3btuGPf/wj7r333hm9LnsthUIxq/vKgh6z/ZpLBR27y0PHb+bo2M0cHbv8OH5UgoUQQghZ/ChYTgghhJCzDAbD2wIMpaWlvDwLIYQQQgghhBCymAnmewcIIYSQucBKgbi9fRgeP8h/stvkXL/97W+xYcOGc5pxhkIhvoS9trZ2XveNEEIICYbtGBk/hAlnG9KZ1HzvDiGEEEKmIZYIYtR+FDb7McQTIeQryiwnhBCy6LHAeEffU2jr+SPiiSCkEg2WN9yOppobsZSl02l4PB6o1WrIZDJs2bIF3//+9/HFL34Rn/3sZ3nN8v/8z//k2ea33XbbfO8uIYSQJWzccRL7T/4EwfAEhAIxqko2Y/2Kj0MkpDrihBBCSL4LhR3Yc+x/4HB38dsWYxOuXP0PUCnMyDeUWU4IIWTR8wWG0d77ZwiFEhSalkEgEOFUzx/hDQxjKRsfH8fmzZvx9NNP89tFRUV48MEHeWb5+9//ftxzzz08kP7rX/8aUql0vneXEELIEsWyyI92PIxwxIlCYzM0qmL0j7yC4bH9871rhBBCCJmCjtNP8pVhJn09TPo6TDhP8YS2fESZ5YQQQha9aMzLM8rNhkbenEuttMLp6UYs5gM05Vgqvvvd755zm9Ui7+7uPue+lpYW/OIXv5jjPSOEEEIuLJEI8bFcpbCgoEAAqUSNbDaDaNw337tGCCGEkCkIRCYgkaggFIr5bfbfodAE8tGsZ5b7fD58/etf50u5V69ezTPTDh8+PNtvQwghhEyZUmmBXKqDLziMVDrBf8plOijzcMkXIYQQQs7FguMqZSGCoXGkUnGEIy6+Skwlp3GcEEILine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1500x500 with 3 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import seaborn as sns\n",
|
||||
"sns.set_theme(style=\"ticks\", palette=\"pastel\")\n",
|
||||
"\n",
|
||||
"def get_angle_colors(positions):\n",
|
||||
" angles = np.arctan2(positions[:, 1], positions[:, 0])\n",
|
||||
" angles_deg = (np.degrees(angles) + 360) % 360\n",
|
||||
" colors = np.zeros((len(positions), 3))\n",
|
||||
" for i, angle in enumerate(angles_deg):\n",
|
||||
" segment = int(angle / 120)\n",
|
||||
" local_angle = angle - segment * 120\n",
|
||||
" if segment == 0: # 0 degrees to 120 degrees (R->G)\n",
|
||||
" colors[i] = [1 - local_angle/120, local_angle/120, 0]\n",
|
||||
" elif segment == 1: # 120 degrees to 240 degrees (G->B)\n",
|
||||
" colors[i] = [0, 1 - local_angle/120, local_angle/120]\n",
|
||||
" else: # 240 degrees to 360° (B->R)\n",
|
||||
" colors[i] = [local_angle/120, 0, 1 - local_angle/120]\n",
|
||||
" return colors\n",
|
||||
"\n",
|
||||
"desired_times = [0.2, 0.6, 0.8,]\n",
|
||||
"time_np = time_points.detach().cpu().numpy()\n",
|
||||
"n_steps = len(time_np)\n",
|
||||
"indices = [np.argmin(np.abs(time_np - t_val)) for t_val in desired_times]\n",
|
||||
"\n",
|
||||
"fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(15, 5))\n",
|
||||
"axes = axes.ravel() # flatten the 2D array for easier indexing\n",
|
||||
"\n",
|
||||
"xx, yy = np.mgrid[0:1:100j, -1:1:100j]\n",
|
||||
"positions = np.vstack([xx.ravel(), yy.ravel()])\n",
|
||||
"\n",
|
||||
"for i, idx in enumerate(indices):\n",
|
||||
" ax = axes[i]\n",
|
||||
" x_t = trajectory[idx].detach().cpu().numpy()\n",
|
||||
" if i == 0:\n",
|
||||
" c = get_angle_colors(x_t)\n",
|
||||
" ax.scatter(x_t[:, 0], x_t[:, 1], alpha=0.5, s=10, color=c)\n",
|
||||
" ax.set_title(f\"t = {time_np[idx]:.2f}\")\n",
|
||||
" ax.grid(True)\n",
|
||||
"\n",
|
||||
"plt.tight_layout(rect=[0, 0.03, 1, 0.95])\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "447fb175-92ea-4014-9e1a-97271cb3d2fb",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's also plot the solution in line with the supervised and differentiable physics variants above:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"id": "fd5fd88a-6a3b-4712-bcd6-8d1c250d5769",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHJCAYAAACG+j24AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAdsJJREFUeJztnQl8FPX5xt9NQgg5IAkJCYEAIRwBI6ByiUdVrLd/rfW+b1tbFFRAq1W8DxDxqPddsV6leFTFSrWtgggqIkIQQjiTQG4CIQfJ/j/Pb3c2M7Mzmw0k2ev5fj7r7hw7OzuJmYf3fd73dTidTqcQQgghhEQoUYE+AUIIIYSQQEIxRAghhJCIhmKIEEIIIRENxRAhhBBCIhqKIUIIIYRENBRDhBBCCIloKIYIIYQQEtFQDBFCCCEkoqEYIoREPOw9S0hkQzFESBhzySWXyPDhww2P/Px8OeaYY+Tuu++WmpqaDvmc4447Tm699dYDPs6CBQvUOW7bts12H2zDPtgXPPnkk2pZ/53x8JfvvvtOrr32Wtvjt5cPPvhAvf+TTz6x3efll1+WvLw82bJli3QWB/o9CIkkYgJ9AoSQzmXkyJFy1113eZabmprk559/lrlz58ratWvlb3/7mzgcDgkV+vTpI2+//bYMGDDAcrv+u/rDu+++K4WFhX4fvy1OOOEEueeee+TDDz+Uk08+2XKff/zjHzJ+/Pj9/gx/ONDvQUgkQTFESJiTmJgoY8aMMawbN26c7NmzR5544gn58ccfvbYHM7GxsT7Pd8iQIZ16/LaIi4uT0047Td577z0VeevVq5dh+5o1a+SXX36R2bNnS2dyoN+DkEiCaTJCIhSky0BxcbF6RmrplltukRtuuEHdRK+44gq1vra2Vh588EE5/vjj5eCDD/bc6M0g4nTfffcpoTV27FiZOXOmVFZWekVhzjrrLHX8UaNGyRlnnGGZTvr+++/lzDPPVOeIz/v444/9Tv+Y02Rff/21nHvuuXLIIYeoc/v973/viQQhtYcozfbt2z3HtDr+xo0b5Y9//KOK5uAY1113nSGaZOa3v/2tuh6ffvqp1zZ8Xs+ePeXEE09Uy9XV1XLnnXfKpEmT1PXFuS5dutTwHpzPU089pa4drhtet7S0yGOPPaZSlLhOeH700UfV59pdp02bNqmf7xFHHKF+BrhOSBOary1+JtgP1wzf+Y477pC6ujrb70tIqEMxREiEUlRUpJ6zs7M963ATTEhIkGeeeUauvvpqqa+vlwsvvFClfLD89NNPy2GHHSa33367PPvss4bj4b1Ivz300ENKCH355ZdyzTXXSHNzs9o+f/58ddOHqHruuedkzpw5KnoBAVZaWmo4FvZDigmfN3ToUJk2bZp8/vnn7f6OW7duleuvv16JBXyn+++/X31veIQgJrDtV7/6laSnp6uUErxUZnbs2CHnnXeeEhKzZs1SEZ3y8nK57LLLlJCxAqIGogLXTc++ffvkn//8p5x++unSvXt3aWhoUMdZvHix+o4QOZmZmepamwURrjfeh2gehNQLL7ygUpx/+MMflAfpggsukJdeekl9Tys2bNigxBQED8QNrj/So/j8b7/91ivV2K9fP3X9r7rqKiV+7Y5LSDjANBkhEVAphZuwBlI3uPnh5oZ/+WsRItCtWzdlrIZIAW+++aZK6bz11ltqX3DUUUep4+FGef7550tycrJan5KSom7G8fHxnmXcqP/73//Kscceq4QJbqwQIBq44eIGjejEqaee6lk/ZcoUtS84+uijlRDB50FItYdVq1YpQYdITkZGhloHsQHxgUgH/DSpqamGlJI5AvLqq69KY2OjvPLKK0o0AZifIT6QYoSYsosOIaKGyFtWVpZah2tRUVEhZ599tlp+//33paCgQN555x0ZPXq05/siYgOx8ve//91zPETbtGgdgOjEzw6fAxDB6dGjhyQlJVmeD4QWvufrr7+uUqcA4g+Rt0ceecQQ7cN3gqAFhx9+uIquQdzefPPN7br+hIQKFEOEhDnLly+Xgw46yLAuKipKpWVg9NWbpwcPHuwRQgCiCYJFE0Ia//d//6dunnoxgGdNCAGkbWJiYtTnQwxp1Wa7du1SaafNmzfLsmXL1DqIDT2nnHKKYRkiCFVj8Dm1BwgMRGAgPk466SQlNCZMmKBSTf4CoQahpAkhTVB98cUXPt+Ha4Qo0kcffeSpVlu4cKEytOMBEP3BcfHz0QtWXC8IFL3naMSIEYbj43sgLYbIHa41hM3FF19sez74WeK4mhAC+PlAhP7lL38xXFuz1wjfF6lEQsIViiFCwhzcaBHtARA+EAd9+/Y13BQ1kCLTg5uxXgRopKWleYSNhnk/CC5Eh7R9UEaO9BcEACJQEF6IsFj1+dGOr9G7d2+1z+7du9v13fv37y9vvPGGPP/880q8ISoCvw4ExNSpU/2qokMqDMdpL/juEClIlUEM4VpCQOlbEODYZWVlXmJVA9s0MaQXmgCpNPy8ED1CFAnCCylFpMAmTpzodSx8vvm6AqwzX1tEmMw/S/ZiIuEMxRAhYQ5umPCw7A+4ESOCY3WT1m74Gmb/DLxCVVVVSsjAnwNBABEEUYIoB6IS8LEgVdTWjRsenejoaHU+eN0eNMMxok+I8sAbBP8NhJhd6bsepJ3MRnAAUQeRpPdcmUFECr4ppBrx2RBf8P3ojz1o0CAlZqzwJcIgUC666CL1QOrtP//5j/peSDEirWXG7trpf5Y7d+60/TxCwhkaqAkhtqByCumRH374wauxIISNPt2EG7A+1bNo0SK1jHQORBGMyxAHEGYQQpqHBkAs6YE/RQPbUJWFlBfK1tsD/D5IDUEIIf0H/8u9995rqKKDqPAFvDpIB+oFEcQHIjMQIL448sgjVYrps88+UwZz9CBCZEoDPp+SkhIlGHFdtAeu5YsvvqgEoB3wa6F6D+D98F5BGCESZxVBw88SkSn9NghWGLrxmfr0KCGRBiNDhBBbcIOFiRpGaJRaI1Lx73//W6VmUGquv7EjwoCoBMy/MDyjqSNKuCFAEBGB9wgVZRAHeN///vc/lbYCe/fuNXzuvHnz1I0a6TxUTEFIwcDcXpAuQtQF5w8/DcQFzOC48UMkAZwLIiYQNmZfDrj88suV1wfiB0ZsiECYz/E99FEeKyC00CIA74f4Mn8HXF+k8WCM/t3vfqe+75IlS1SlGM4Xn2UHxA2qyBBBg6cLVW84PgQWTOFmIzh+XhCfl156qSdKh8+GsR3Ci5BIhmKIEGILvCN//etflVH38ccfV1EFeH1Qoq5VRGnAh4OeRBAeEBsQCtOnT/f4clANhvfBM4PtaI4IUfHAAw/IihUrDL2BUIWFaimk6IYNG6bEAW7y7QWpMKSOYBC+6aablMBCBRZEBL6HJkgghDTBZzZvQ6BAEMKTo507ol3o8WNuqGgFrhNaCSCdZv4O8AFBIOL64vi4fhCNqNq68sorfR73xhtvVOcCYYrvh5QbPEp2FV/wE+F7QKTedttt6ueCyB4EKaJfhEQyDiddcYQQQgiJYOgZIoQQQkhEQzFECCGEkIiGYogQQgghEQ3FECGEEEIiGoohQgghhEQ0FEOEEEIIiWjYZ8gP0IMDHWytZjQRQgghJDhBM1j040IvM19QDPlBQ0ODatZGCCGEkNABI4H8aadIMeQHffr0Uc+LFy8O9KkQQgghxE8mT57s1370DBFCCCEkoqEYIoQQQkhEQzFECCGEkIiGYogQQgghEQ0N1B0Mqs6ampoCfRokjOjWrZtER0cH+jQIISRsoRjqIFC6V1paKtXV1YE+FRKGJCcnS2ZmpjgcjkCfCiGEhB0UQx2EJoRQhh8fH8+bFukwkV1XVyc7d+5Uy3379g30KRFCSNhBMdRBqTFNCPXu3TvQp0PCjB49eqhnCCL8jjFlRgghHQsN1B2A5hFCRIiQzkD73aIfjRBCOh6KoQ6EqTHSWfB3ixBCOg+KIUIIIYRENBRDhBBCCOk46raJ7PjC9RwiUAwRA8cdd5wMHz7c88jPz5djjjlG7rrrLqmsrGxXFdQ//vEPqaiokK4+/yeffNLv/YuLi+Wf//xnp54TIYREDIUviSwcKLL4ONczlkMAiiHixZVXXilfffWVenzyySfy5z//WZYtWyYXX3yx1NbW+nWM5cuXy6233ip79+6VruS9995T5+8vM2fOlP/973+dek6EEBIR1G0TWXatiLS4V7SIfHtdSESIKIaCFFQN7d69OyDVQ6hcSk9PV4/s7GyZPHmyvPzyy1JSUiIvvvii35GhQJCamioJCQkB+WxCCIloatfrhJAbZ7NI7QYJdiiGghCko9atWyebNm1Sz+1JT3UWWVlZ8utf/9qTUvrll1/kuuuuk3HjxqlUmiaYAKJIl156qXqN9QsWLFCv3333XTn99NNl1KhRMmbMGLnwwgvlp59+MqS4nn76abnqqqvUPvg8vEfPDz/8oI592GGHyYQJE+S2226TqqoqyzQZni+//HJ5/vnn5eijj5aDDz5YRbcKCwvV9ksuuUS+/fZblc7D+8CqVavUeR1yyCHqu02ZMkWl0gghhLRB0lBvWeGIFkkaIsEOxVCQgUiQ+eaL5WDoLzNs2DDZunWrilghFYUREW+99ZZ89NFHctJJJ8nDDz8sa9euVUJCEyQQM6eccor861//knvuuUeuvvpqlXp79dVXpaGhQe644w7DZ0AM4f0LFy6Uiy66SO688075+OOPPUIFAmbo0KHyzjvvyOOPPy4//vijEk9ofGnFihUr5LvvvlOC6M0331Qeprvvvlttwznis04++WSVXsMxNIH3wQcfqHPEtf/Tn/7U6deWEEJCnvj+IhOedwkggOfxz7nWBznsQB1kQCDYrcfAzkDSs2dP9VxTU6OiMxArWkrqhhtuUCk0RLJGjBghvXr18qSt4uLilHC6//775f/+7//U+n79+snZZ5+tBJKeI488Uv74xz+q14MLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 640x480 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Results\n",
|
||||
"plt.plot(X,Y,'.',label='Datapoints', color=\"lightgray\")\n",
|
||||
"plt.plot(trajectory[-1][:,0].detach(),trajectory[-1][:,1].detach(),'.',label='Flow Matching', color=\"orange\") \n",
|
||||
"plt.xlabel('x')\n",
|
||||
"plt.ylabel('y')\n",
|
||||
"plt.title('Probabilistic Version')\n",
|
||||
"plt.legend()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2ad4c561-6068-418a-bb53-1a3bf2ebcbdf",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"As promised, this approach actually resolves both \"modes\" of the solution in the form of points above and below the x-axis. It's still a bit noisy, but this could be alleviated by improving the learning setup, e.g., a larger network would help.\n",
|
||||
"\n",
|
||||
"An obvious question here also is: we're back to training only with data, how about integrating the physics? That's an obvious point for improvements, and we'll address diffusion-based methods with physical constraints in more detail in a later section. As an outlook: physics-priors can help especially to drive the somewhat noisy output of a neural network towards an accurate solution.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Discussion\n",
|
||||
"\n",
|
||||
"It's a very simple example, but it very clearly shows a failure case for supervised learning. While it might seem very artificial at first sight, many practical PDEs exhibit a variety of these modes, and it's often not clear where (and how many) exist in the solution manifold we're interested in. Using supervised learning is very dangerous in such cases. We might unknowingly get an average of these different modes.\n",
|
||||
@@ -406,7 +701,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "useful-special",
|
||||
"id": "1c33bc02-4f52-4fce-a40c-f72cd9c9ac9b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
@@ -415,7 +710,7 @@
|
||||
"\n",
|
||||
"For the simple DP example above:\n",
|
||||
"\n",
|
||||
"- This notebook is intentionally using a very simple setup. Change the training setup and NN above to obtain a higher-quality solution such as the green one shown in the very first image at the top. \n",
|
||||
"- This notebook is intentionally using a very simple setup. You can change the training setup or network architectures to improve the solutions. E.g., provide more samples around zero to improve the solution near the origin.\n",
|
||||
"\n",
|
||||
"- Or try extending the setup to a 2D case, i.e. a paraboloid. Given the function $\\mathcal P:(y_1,y_2)\\to y_1^2+y_2^2$, find an inverse function $f$ such that $\\mathcal P(f(x)) = x$ for all $x$ in $[0,1]$.\n",
|
||||
"\n",
|
||||
@@ -425,7 +720,7 @@
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
@@ -439,7 +734,7 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.5"
|
||||
"version": "3.12.9"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 95 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 200 KiB After Width: | Height: | Size: 252 KiB |
Binary file not shown.
Executable → Regular
BIN
Binary file not shown.
Reference in new issue
Block a user