daviddoji/18065
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

draft SGD notebook

Browse Source
This commit is contained in:
Steven G. Johnson committed 2023-03-23 16:05:28 -04:00
1 parent c98bda5c76
commit 539ecc3331
1 file changed
+735
Download Patch File Download Diff File Expand all files Collapse all files
notes/Stochastic-Gradient-Descent.ipynb
+735
View File
@@ -0,0 +1,735 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "33569260",
"metadata": {},
"outputs": [],
"source": [
"using PyPlot, LinearAlgebra"
]
},
{
"cell_type": "markdown",
"id": "baafb243",
"metadata": {},
"source": [
"## A simple least-squares fitting problem\n",
"\n",
"Let's fit $m$ data points $(a_i, b_i)$ to a straight line $a(b) = x_1 + x_2 b$. With a least-square fit, minimizing\n",
"$$f(x) = \\Vert b - Ax \\Vert^2/m =\\frac{1}{m} \\sum_i \\underbrace{(b_i - (Ax)_i)^2}_{f_i(x)}\\, ,$$\n",
"the solution is simply $\\hat{x} = A \\setminus b = (A^T A)^{-1} A^T b$ where\n",
"\n",
"$$\n",
"A = \\begin{pmatrix} 1 & a_1 \\\\ 1 & a_2 \\\\ \\vdots & \\vdots \\\\ 1 & a_m \\end{pmatrix}\n",
"$$\n",
"\n",
"Let's do that for 1000 data points given by a line $1 + 2a$ plus some noise:"
]
},
{
"cell_type": "code",
"execution_count": 77,
"id": "7413cb8b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"2-element Vector{Float64}:\n",
" 0.909278704461504\n",
" 2.0709823897437754"
]
},
"execution_count": 77,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m = 1000\n",
"a = randn(m)\n",
"b = @. 1 + 2a + randn()\n",
"A = [one.(a) a]\n",
"x̂ = A \\ b"
]
},
{
"cell_type": "code",
"execution_count": 78,
"id": "f28865a2",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAGxCAYAAABfrt1aAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAACHtElEQVR4nO3dd3hUVfoH8O+dkYQWAhFCCx2sKCggRVyKCmtBLCAWMC4RAUFFBAHdn2INBEVZFKREUFEkuiCiq4gaAaUFFRuCiJQEUEQwgdAz5/fH8U5unzuTmcwk+X6eZx7MnVvO3GT3vnPOe96jCCEEiIiIiKLAE+0GEBERUcXFQISIiIiihoEIERERRQ0DESIiIooaBiJEREQUNQxEiIiIKGoYiBAREVHUMBAhIiKiqGEgQkRERFHDQIQojLp3747u3bv7f965cycURcH8+fOj1qaSuPPOO9G0adNoNwNvvvkmXnjhhWg3o0w4ePAgbrnlFiQnJ0NRFFx//fUAAEVRMHHiRP9+mzdvxsSJE7Fz586otJNIdUa0G0BUntWvXx9r165FixYtot2UMu3NN9/EDz/8gFGjRkW7KTHvySefxJIlS/DKK6+gRYsWSEpKAgCsXbsWKSkp/v02b96Mxx9/HN27d4+JYJMqLgYiREE4duwYKleuDEVRXO0fHx+PTp06RbhV1o4ePYqqVatG5doUPCEEjh8/jipVqpToPD/88ANatGiB22+/Xbc9Wn+HRIFwaIYqpC+++AKXX345EhISULVqVXTp0gUffPCBbp/58+dDURR8/PHHGDx4MOrUqYOqVavixIkTEEIgIyMDTZo0QeXKlXHxxRfjww8/NF3Hamhm4sSJUBQFP/74I2699VYkJiaibt26GDx4MPLz83XHv/TSS/jHP/6B5ORkVKtWDRdccAEyMjJw6tQp3X7du3dH69atsWrVKnTp0gVVq1bF4MGDkZaWhqSkJBw9etTUtp49e+L8888P+t4JITBjxgy0bdsWVapUQa1atdCvXz/8+uuvuv1WrFiBvn37IiUlBZUrV0bLli0xdOhQHDhwQLffH3/8gbvvvhuNGjVCfHw86tSpg0svvRSffPKJ/7N98MEH2LVrFxRF8b8Cefvtt9GxY0ckJiaiatWqaN68OQYPHqzbZ8uWLfjnP/+JqlWronbt2hg2bBiWLVsGRVHw+eef+/dr2rQp7rzzTtM1jENxx48fx4MPPoi2bdsiMTERSUlJ6Ny5M5YuXWo6VlEUjBw5Ei+//DLOPfdcxMfH49VXXwUAbNu2DbfddhuSk5MRHx+Pc889Fy+99JLj51X/1j755BP89NNP/vukfg7t0Mz8+fPRv39/AECPHj38+5bVIUQq29gjQhXOypUrceWVV+LCCy9EZmYm4uPjMWPGDPTp0wcLFy7EgAEDdPsPHjwY11xzDV5//XUUFhaiUqVKePzxx/H4448jLS0N/fr1Q25uLoYMGYKioiKcffbZrtpx0003YcCAAUhLS8P333+PCRMmAABeeeUV/z7bt2/HbbfdhmbNmiEuLg7ffvstnn76aWzZskW3HwDs27cPAwcOxEMPPYRnnnkGHo8HNWvWxCuvvII333wTd911l3/fzZs3Izs7O+DDzcrQoUMxf/583HfffZg8eTIOHjyIJ554Al26dMG3336LunXr+tveuXNn3HXXXUhMTMTOnTsxdepUdO3aFd9//z0qVaoEABg0aBC+/vprPP300zjrrLPw119/4euvv8aff/4JAJgxYwbuvvtubN++HUuWLHHVxrVr12LAgAEYMGAAJk6ciMqVK2PXrl347LPP/Pv8/vvv6NatGypVqoQZM2agbt26eOONNzBy5Mig74nqxIkTOHjwIMaMGYOGDRvi5MmT+OSTT3DjjTdi3rx5uOOOO3T7v/vuu1i9ejUeffRR1KtXD8nJydi8eTO6dOmCxo0b47nnnkO9evWwfPly3HfffThw4AAee+wxy2urw4D33HMP8vPz8cYbbwAAzjvvPNO+11xzDZ555hk8/PDDeOmll3DxxRcDAIcQKToEUQXTqVMnkZycLA4fPuzfdvr0adG6dWuRkpIifD6fEEKIefPmCQDijjvu0B1/6NAhUblyZXHDDTfotn/55ZcCgOjWrZt/244dOwQAMW/ePP+2xx57TAAQGRkZuuPvueceUblyZf/1jYqKisSpU6fEa6+9Jrxerzh48KD/vW7dugkA4tNPPzUd161bN9G2bVvdtuHDh4saNWro7oGV1NRU0aRJE//Pa9euFQDEc889p9svNzdXVKlSRTz00EOW5/H5fOLUqVNi165dAoBYunSp/73q1auLUaNGObbjmmuu0bUjkGeffVYAEH/99ZftPuPGjROKoohNmzbptl955ZUCgMjOzvZva9KkiUhNTTWdo1u3brrft9Hp06fFqVOnRFpamrjooot07wEQiYmJut+jEEL07t1bpKSkiPz8fN32kSNHisqVK5v2t2rT+eefb9oOQDz22GP+n99++23T5ySKBg7NUIVSWFiI9evXo1+/fqhevbp/u9frxaBBg5CXl4etW7fqjrnpppt0P69duxbHjx83jcF36dIFTZo0cd2W6667TvfzhRdeiOPHj2P//v3+bd988w2uu+46nHnmmfB6vahUqRLuuOMOFBUV4eeff9YdX6tWLfTs2dN0nfvvvx+bNm3Cl19+CQAoKCjA66+/jtTUVN09cOP999+HoigYOHAgTp8+7X/Vq1cPbdq00Q1n7N+/H8OGDUOjRo1wxhlnoFKlSv7789NPP/n3u+SSSzB//nw89dRTWLdunWnYyUlRUZGuHT6fDwDQoUMHAMDNN9+MrKws7Nmzx3RsdnY2zj//fLRp00a3/bbbbnN9fStvv/02Lr30UlSvXt3/uTMzM3WfWdWzZ0/UqlXL//Px48fx6aef4oYbbkDVqlV1n+3qq6/G8ePHsW7duhK1jyjWMBChCuXQoUMQQqB+/fqm9xo0aAAA/iEBlXFf9f169eqZzmG1zc6ZZ56p+zk+Ph6ATIgFgN27d+Oyyy7Dnj17MG3aNKxevRo5OTn+4RR1P7t2qvr27YumTZv6j5s/fz4KCwsxYsQI121V/f777xBCoG7duqhUqZLutW7dOn/+h8/nQ69evbB48WI89NBD+PTTT7Fhwwb/Q1Tb9kWLFiE1NRVz585F586dkZSUhDvuuAO//fZbwPZcfvnlujaoOSD/+Mc/8O677+L06dO44447kJKSgtatW2PhwoX+Y//8888S/w6NFi9ejJtvvhkNGzbEggULsHbtWuTk5GDw4ME4fvy4aX+rv63Tp09j+vTppvt79dVXA4Apx4aorGOOCFUotWrVgsfjwb59+0zv7d27FwBQu3Zt3XZjYqQaQFg9KH/77bewTYV89913UVhYiMWLF+t6WjZt2mS5v10Cp8fjwYgRI/Dwww/jueeew4wZM3D55Ze7zmXRql27NhRFwerVq/2Bk5a67YcffsC3336L+fPnIzU11f/+L7/8YnnOF154AS+88AJ2796N9957D+PHj8f+/fvx0UcfObZn1qxZOHz4sO5cqr59+6Jv3744ceIE1q1bh/T0dNx2221o2rQpOnfujDPPPNP2d2hUuXJlnDhxwrT9wIEDumsuWLAAzZo1w6JFi3S/D6tjAfPvrFatWv7eObtAsVmzZpbbicoqBiJUoVSrVg0dO3bE4sWL8eyzz/qnSvp8PixYsAApKSk466yzHM/RqVMnVK5cGW+88YZu2GbNmjXYtWtX2AIR9SGlfeALITBnzpygz3XXXXdh4sSJuP3227F161ZMnjw5pDZde+21mDRpEvbs2YObb77Zdj+rtgMycHDSuHFjjBw5Ep9++ql/KEk9j7EHCICrYCo+Ph7dunVDzZo1sXz5cnzzzTfo3LkzevTogYyMDHz77be64Zk333zTdI6mTZviu+++0237+eefsXXrVl0goigK4uLidAHGb7/9ZjlrxkrVqlXRo0cPfPPNN7jwwgsRFxfn6rhQGHvgiKKFgQhVOOnp6bjyyivRo0cPjBkzBnFxcZgxYwZ++OEHLFy4MODU0Fq1amHMmDF46qmncNddd6F///7Izc3FxIkTS9Stb3TllVciLi4Ot956Kx566CEcP34cM2fOxKFDh4I+V82aNXHHHXdg5syZaNKkCfr06RNSmy699FLcfffd+Ne//oWNGzfiH//4B6pVq4Z9+/bhiy++wAUXXIDhw4fjnHPOQYsWLTB+/HgIIZCUlIRly5ZhxYoVuvPl5+ejR48euO2223DOOecgISEBOTk5+Oijj3DjjTf697vggguwePFizJw5E+3atYPH40H79u1t2/noo48iLy8Pl19+OVJSUvDXX39h2rRpqFSpErp16wYAGDVqFF555RVcc801eOqpp/yzZrZs2WI636BBgzBw4EDcc889uOmmm7Br1y5kZGSgTp06uv2uvfZaLF68GPfcc49/NtWTTz6J+vXrY9u2ba7u8bRp09C1a1dcdtllGD58OJo2bYrDhw/jl19+wbJly3Qzf0qidevWAIDZs2cjISEBlStXRrNmzUxDhkQRF91cWaLoWL16tejZs6eoVq2aqFKliujUqZNYtmyZbh911kxOTo7peJ/PJ9LT00WjRo1EXFycuPDCC8WyZctMsyicZs388ccfltfbsWOHf9uyZctEmzZtROXKlUXDhg3F2LFjxYcffmia7WA3U0Lr888/FwDEpEmTAt+gvxlnzaheeeUV0bFjR//9a9GihbjjjjvExo0b/fts3rxZXHnllSIhIUHUqlVL9O/fX+zevVs3e+P48eNi2LBh4sILLxQ1atQQVapUEWeffbZ47LHHRGFhof9cBw8eFP369RM1a9YUiqKIQP/X9f7774urrrpKNGzYUMTFxYnk5GRx9dVXi9WrV+v2U9tYuXJlkZSUJNLS0sTSpUtN99fn84mMjAzRvHlzUblyZdG+fXvLine truncated
"text/plain": [
"Figure(PyObject <Figure size 640x480 with 1 Axes>)"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"PyObject <matplotlib.legend.Legend object at 0x7fa68f62be50>"
]
},
"execution_count": 78,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"title(\"ordinary least-square fit\")\n",
"plot(a, b, \"b.\")\n",
"plot(a, x̂[1] .+ x̂[2] .* a, \"r-\")\n",
"legend([L\"data $1+2a+\\mathrm{noise}$\", L\"fit $%$(round(x̂[1],digits=3)) + %$(round(x̂[2],digits=3))a$\"])"
]
},
{
"cell_type": "markdown",
"id": "c168087c",
"metadata": {},
"source": [
"## Gradient descent\n",
"\n",
"We can instead solve this problem by gradient descent with a fixed stepsize $s$. At each iteration\n",
"$$\n",
"x^{(k+1)} = x^{(k)} - s \\nabla f^{(k)}\n",
"$$\n",
"where for $f(x) = \\frac{1}{m}\\Vert b - Ax \\Vert^2 = \\frac{1}{m}(b-Ax)^T (b-Ax)$ we have $\\nabla f = \\frac{2}{m}A^T(Ax - b)$. We'll start with an initial \"guess\" of $x^{(1)} = [0,0]$.\n",
"\n",
"This is sometimes called **\"full batch\" gradient descent\"**: we are looking at the **whole \"batch\"** of training data at once.\n",
"\n",
"We'll use the [Interact.jl package](https://github.com/JuliaGizmos/Interact.jl) to visualize the progress of this algorithm as we vary the stepsize $s$.\n",
"\n",
"For goodd convergence, we should typically have $s < 1/L$ where $L \\approx 2$ is the Lipschitz constant of $f$, corresponding to the largest eigenvalue of the Hessian $2A^T A / m$. So, convenience we'll specify $sL$ instead of $s$ — for $sL > 1$ we'll see that it \"overshoots\" the optimum and oscillates."
]
},
{
"cell_type": "code",
"execution_count": 81,
"id": "4b186c51",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"L = (2 / m) * maximum(eigvals(A' * A)) = 2.1850688743919733\n"
]
},
{
"data": {
"application/vnd.webio.node+json": {
"children": [
{
"children": [
{
"children": [
{
"children": [
{
"children": [
{
"children": [
"s × L"
],
"instanceArgs": {
"namespace": "html",
"tag": "label"
},
"nodeType": "DOM",
"props": {
"className": "interact ",
"style": {
"padding": "5px 10px 0px 10px"
}
},
"type": "node"
}
],
"instanceArgs": {
"namespace": "html",
"tag": "div"
},
"nodeType": "DOM",
"props": {
"className": "interact-flex-row-left"
},
"type": "node"
},
{
"children": [
{
"children": [],
"instanceArgs": {
"namespace": "html",
"tag": "input"
},
"nodeType": "DOM",
"props": {
"attributes": {
"data-bind": "numericValue: index, valueUpdate: 'input', event: {change: function (){this.changes(this.changes()+1)}}",
"orient": "horizontal",
"type": "range"
},
"className": "slider slider is-fullwidth",
"max": 40,
"min": 1,
"step": 1,
"style": {}
},
"type": "node"
}
],
"instanceArgs": {
"namespace": "html",
"tag": "div"
},
"nodeType": "DOM",
"props": {
"className": "interact-flex-row-center"
},
"type": "node"
},
{
"children": [
{
"children": [],
"instanceArgs": {
"namespace": "html",
"tag": "p"
},
"nodeType": "DOM",
"props": {
"attributes": {
"data-bind": "text: formatted_val"
}
},
"type": "node"
}
],
"instanceArgs": {
"namespace": "html",
"tag": "div"
},
"nodeType": "DOM",
"props": {
"className": "interact-flex-row-right"
},
"type": "node"
}
],
"instanceArgs": {
"namespace": "html",
"tag": "div"
},
"nodeType": "DOM",
"props": {
"className": "interact-flex-row interact-widget"
},
"type": "node"
}
],
"instanceArgs": {
"handlers": {
"changes": [
"(function (val){return (val!=this.model[\"changes\"]()) ? (this.valueFromJulia[\"changes\"]=true, this.model[\"changes\"](val)) : undefined})"
],
"index": [
"(function (val){return (val!=this.model[\"index\"]()) ? (this.valueFromJulia[\"index\"]=true, this.model[\"index\"](val)) : undefined})"
]
},
"id": "7102712321243811129",
"imports": {
"data": [
{
"name": "knockout",
"type": "js",
"url": "/assetserver/2df50fa7f121fedbb109a97bba894e56aef8c852-knockout.js"
},
{
"name": "knockout_punches",
"type": "js",
"url": "/assetserver/bb2c111220e03062c2973eb306d97c8fc96857d6-knockout_punches.js"
},
{
"name": null,
"type": "js",
"url": "/assetserver/dc651e248070c73d6a4eaa2766193d4a09a13f05-all.js"
},
{
"name": null,
"type": "css",
"url": "/assetserver/0318b018d0ff756043bf1ee25f56ab5c74ef7889-style.css"
},
{
"name": null,
"type": "css",
"url": "/assetserver/f798281fed8908611ae453136bda5c1a2e6bffd7-bulma_confined.min.css"
}
],
"type": "async_block"
},
"mount_callbacks": [
"function () {\n var handler = (function (ko, koPunches) {\n ko.punches.enableAll();\n ko.bindingHandlers.numericValue = {\n init: function(element, valueAccessor, allBindings, data, context) {\n var stringified = ko.observable(ko.unwrap(valueAccessor()));\n stringified.subscribe(function(value) {\n var val = parseFloat(value);\n if (!isNaN(val)) {\n valueAccessor()(val);\n }\n });\n valueAccessor().subscribe(function(value) {\n var str = JSON.stringify(value);\n if ((str == \"0\") && ([\"-0\", \"-0.\"].indexOf(stringified()) >= 0))\n return;\n if ([\"null\", \"\"].indexOf(str) >= 0)\n return;\n stringified(str);\n });\n ko.applyBindingsToNode(\n element,\n {\n value: stringified,\n valueUpdate: allBindings.get('valueUpdate'),\n },\n context,\n );\n }\n };\n var json_data = {\"formatted_vals\":[\"0.05\",\"0.1\",\"0.15\",\"0.2\",\"0.25\",\"0.3\",\"0.35\",\"0.4\",\"0.45\",\"0.5\",\"0.55\",\"0.6\",\"0.65\",\"0.7\",\"0.75\",\"0.8\",\"0.85\",\"0.9\",\"0.95\",\"1.0\",\"1.05\",\"1.1\",\"1.15\",\"1.2\",\"1.25\",\"1.3\",\"1.35\",\"1.4\",\"1.45\",\"1.5\",\"1.55\",\"1.6\",\"1.65\",\"1.7\",\"1.75\",\"1.8\",\"1.85\",\"1.9\",\"1.95\",\"2.0\"],\"changes\":WebIO.getval({\"name\":\"changes\",\"scope\":\"7102712321243811129\",\"id\":\"1344084652692618713\",\"type\":\"observable\"}),\"index\":WebIO.getval({\"name\":\"index\",\"scope\":\"7102712321243811129\",\"id\":\"10488540316168852659\",\"type\":\"observable\"})};\n var self = this;\n function AppViewModel() {\n for (var key in json_data) {\n var el = json_data[key];\n this[key] = Array.isArray(el) ? ko.observableArray(el) : ko.observable(el);\n }\n \n [this[\"formatted_val\"]=ko.computed( function(){\n return this.formatted_vals()[parseInt(this.index())-(1)];\n }\n,this)]\n [this[\"changes\"].subscribe((function (val){!(this.valueFromJulia[\"changes\"]) ? (WebIO.setval({\"name\":\"changes\",\"scope\":\"7102712321243811129\",\"id\":\"1344084652692618713\",\"type\":\"observable\"},val)) : undefined; return this.valueFromJulia[\"changes\"]=false}),self),this[\"index\"].subscribe((function (val){!(this.valueFromJulia[\"index\"]) ? (WebIO.setval({\"name\":\"index\",\"scope\":\"7102712321243811129\",\"id\":\"10488540316168852659\",\"type\":\"observable\"},val)) : undefined; return this.valueFromJulia[\"index\"]=false}),self)]\n \n }\n self.model = new AppViewModel();\n self.valueFromJulia = {};\n for (var key in json_data) {\n self.valueFromJulia[key] = false;\n }\n ko.applyBindings(self.model, self.dom);\n}\n);\n (WebIO.importBlock({\"data\":[{\"name\":\"knockout\",\"type\":\"js\",\"url\":\"/assetserver/2df50fa7f121fedbb109a97bba894e56aef8c852-knockout.js\"},{\"name\":\"knockout_punches\",\"type\":\"js\",\"url\":\"/assetserver/bb2c111220e03062c2973eb306d97c8fc96857d6-knockout_punches.js\"}],\"type\":\"async_block\"})).then((imports) => handler.apply(this, imports));\n}\n"
],
"observables": {
"changes": {
"id": "1344084652692618713",
"sync": false,
"value": 0
},
"index": {
"id": "10488540316168852659",
"sync": true,
"value": 6
}
},
"systemjs_options": null
},
"nodeType": "Scope",
"props": {},
"type": "node"
}
],
"instanceArgs": {
"namespace": "html",
"tag": "div"
},
"nodeType": "DOM",
"props": {
"className": "field interact-widget"
},
"type": "node"
},
{
"children": [
{
"children": [
{
"children": [
{
"children": [
{
"children": [
"# iterations"
],
"instanceArgs": {
"namespace": "html",
"tag": "label"
},
"nodeType": "DOM",
"props": {
"className": "interact ",
"style": {
"padding": "5px 10px 0px 10px"
}
},
"type": "node"
}
],
"instanceArgs": {
"namespace": "html",
"tag": "div"
},
"nodeType": "DOM",
"props": {
"className": "interact-flex-row-left"
},
"type": "node"
},
{
"children": [
{
"children": [],
"instanceArgs": {
"namespace": "html",
"tag": "input"
},
"nodeType": "DOM",
"props": {
"attributes": {
"data-bind": "numericValue: index, valueUpdate: 'input', event: {change: function (){this.changes(this.changes()+1)}}",
"orient": "horizontal",
"type": "range"
},
"className": "slider slider is-fullwidth",
"max": 100,
"min": 10,
"step": 1,
"style": {}
},
"type": "node"
}
],
"instanceArgs": {
"namespace": "html",
"tag": "div"
},
"nodeType": "DOM",
"props": {
"className": "interact-flex-row-center"
},
"type": "node"
},
{
"children": [
{
"children": [],
"instanceArgs": {
"namespace": "html",
"tag": "p"
},
"nodeType": "DOM",
"props": {
"attributes": {
"data-bind": "text: formatted_val"
}
},
"type": "node"
}
],
"instanceArgs": {
"namespace": "html",
"tag": "div"
},
"nodeType": "DOM",
"props": {
"className": "interact-flex-row-right"
},
"type": "node"
}
],
"instanceArgs": {
"namespace": "html",
"tag": "div"
},
"nodeType": "DOM",
"props": {
"className": "interact-flex-row interact-widget"
},
"type": "node"
}
],
"instanceArgs": {
"handlers": {
"changes": [
"(function (val){return (val!=this.model[\"changes\"]()) ? (this.valueFromJulia[\"changes\"]=true, this.model[\"changes\"](val)) : undefined})"
],
"index": [
"(function (val){return (val!=this.model[\"index\"]()) ? (this.valueFromJulia[\"index\"]=true, this.model[\"index\"](val)) : undefined})"
]
},
"id": "13636877548954652155",
"imports": {
"data": [
{
"name": "knockout",
"type": "js",
"url": "/assetserver/2df50fa7f121fedbb109a97bba894e56aef8c852-knockout.js"
},
{
"name": "knockout_punches",
"type": "js",
"url": "/assetserver/bb2c111220e03062c2973eb306d97c8fc96857d6-knockout_punches.js"
},
{
"name": null,
"type": "js",
"url": "/assetserver/dc651e248070c73d6a4eaa2766193d4a09a13f05-all.js"
},
{
"name": null,
"type": "css",
"url": "/assetserver/0318b018d0ff756043bf1ee25f56ab5c74ef7889-style.css"
},
{
"name": null,
"type": "css",
"url": "/assetserver/f798281fed8908611ae453136bda5c1a2e6bffd7-bulma_confined.min.css"
}
],
"type": "async_block"
},
"mount_callbacks": [
"function () {\n var handler = (function (ko, koPunches) {\n ko.punches.enableAll();\n ko.bindingHandlers.numericValue = {\n init: function(element, valueAccessor, allBindings, data, context) {\n var stringified = ko.observable(ko.unwrap(valueAccessor()));\n stringified.subscribe(function(value) {\n var val = parseFloat(value);\n if (!isNaN(val)) {\n valueAccessor()(val);\n }\n });\n valueAccessor().subscribe(function(value) {\n var str = JSON.stringify(value);\n if ((str == \"0\") && ([\"-0\", \"-0.\"].indexOf(stringified()) >= 0))\n return;\n if ([\"null\", \"\"].indexOf(str) >= 0)\n return;\n stringified(str);\n });\n ko.applyBindingsToNode(\n element,\n {\n value: stringified,\n valueUpdate: allBindings.get('valueUpdate'),\n },\n context,\n );\n }\n };\n var json_data = {\"formatted_vals\":[\"10\",\"11\",\"12\",\"13\",\"14\",\"15\",\"16\",\"17\",\"18\",\"19\",\"20\",\"21\",\"22\",\"23\",\"24\",\"25\",\"26\",\"27\",\"28\",\"29\",\"30\",\"31\",\"32\",\"33\",\"34\",\"35\",\"36\",\"37\",\"38\",\"39\",\"40\",\"41\",\"42\",\"43\",\"44\",\"45\",\"46\",\"47\",\"48\",\"49\",\"50\",\"51\",\"52\",\"53\",\"54\",\"55\",\"56\",\"57\",\"58\",\"59\",\"60\",\"61\",\"62\",\"63\",\"64\",\"65\",\"66\",\"67\",\"68\",\"69\",\"70\",\"71\",\"72\",\"73\",\"74\",\"75\",\"76\",\"77\",\"78\",\"79\",\"80\",\"81\",\"82\",\"83\",\"84\",\"85\",\"86\",\"87\",\"88\",\"89\",\"90\",\"91\",\"92\",\"93\",\"94\",\"95\",\"96\",\"97\",\"98\",\"99\",\"100\"],\"changes\":WebIO.getval({\"name\":\"changes\",\"scope\":\"13636877548954652155\",\"id\":\"8228083760766230794\",\"type\":\"observable\"}),\"index\":WebIO.getval({\"name\":\"index\",\"scope\":\"13636877548954652155\",\"id\":\"16764597858456387996\",\"type\":\"observable\"})};\n var self = this;\n function AppViewModel() {\n for (var key in json_data) {\n var el = json_data[key];\n this[key] = Array.isArray(el) ? ko.observableArray(el) : ko.observable(el);\n }\n \n [this[\"formatted_val\"]=ko.computed( function(){\n return this.formatted_vals()[parseInt(this.index())-(10)];\n }\n,this)]\n [this[\"changes\"].subscribe((function (val){!(this.valueFromJulia[\"changes\"]) ? (WebIO.setval({\"name\":\"changes\",\"scope\":\"13636877548954652155\",\"id\":\"8228083760766230794\",\"type\":\"observable\"},val)) : undefined; return this.valueFromJulia[\"changes\"]=false}),self),this[\"index\"].subscribe((function (val){!(this.valueFromJulia[\"index\"]) ? (WebIO.setval({\"name\":\"index\",\"scope\":\"13636877548954652155\",\"id\":\"16764597858456387996\",\"type\":\"observable\"},val)) : undefined; return this.valueFromJulia[\"index\"]=false}),self)]\n \n }\n self.model = new AppViewModel();\n self.valueFromJulia = {};\n for (var key in json_data) {\n self.valueFromJulia[key] = false;\n }\n ko.applyBindings(self.model, self.dom);\n}\n);\n (WebIO.importBlock({\"data\":[{\"name\":\"knockout\",\"type\":\"js\",\"url\":\"/assetserver/2df50fa7f121fedbb109a97bba894e56aef8c852-knockout.js\"},{\"name\":\"knockout_punches\",\"type\":\"js\",\"url\":\"/assetserver/bb2c111220e03062c2973eb306d97c8fc96857d6-knockout_punches.js\"}],\"type\":\"async_block\"})).then((imports) => handler.apply(this, imports));\n}\n"
],
"observables": {
"changes": {
"id": "8228083760766230794",
"sync": false,
"value": 0
},
"index": {
"id": "16764597858456387996",
"sync": true,
"value": 20
}
},
"systemjs_options": null
},
"nodeType": "Scope",
"props": {},
"type": "node"
}
],
"instanceArgs": {
"namespace": "html",
"tag": "div"
},
"nodeType": "DOM",
"props": {
"className": "field interact-widget"
},
"type": "node"
},
{
"children": [
{
"children": [],
"instanceArgs": {
"id": "15813886131965824408",
"name": "obs-node"
},
"nodeType": "ObservableNode",
"props": {},
"type": "node"
}
],
"instanceArgs": {
"handlers": {},
"id": "3013413565185011723",
"imports": {
"data": [],
"type": "async_block"
},
"mount_callbacks": [],
"observables": {
"obs-node": {
"id": "15813886131965824408",
"sync": false,
"value": {
"children": [
{
"children": [],
"instanceArgs": {
"namespace": "html",
"tag": "div"
},
"nodeType": "DOM",
"props": {
"setInnerHtml": "<img src='data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA04AAAGICAYAAAB2jrABAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAADxzUlEQVR4nOzdd3RUVdfA4d+k9056IAm9Q+hIIHTpUgREBQQLgiLyYkHeVwQL9g8sgEpTlCISUIp0AqG30FuAACG9kN5n7vfHJNFAyiSZyZ1JzrPWrEUmd+7ZMySZu+ecvY9CkiQJQRAEQRAEQRAEoUxGcgcgCIIgCIIgCIKg70TiJAiCIAiCIAiCUAGROAmCIAiCIAiCIFRAJE6CIAiCIAiCIAgVEImTIAiCIAiCIAhCBUTiJAiCIAiCIAiCUAGROAmCIAiCIAiCIFRAJE6CIAiCIAiCIAgVEImTIAiCIAiCIAhCBUTiJFSbr68vkydPLv46JCQEhUJBSEiITsddunQpa9asqfZ57t69i0Kh0Mq59Mm6detYvHixxscHBQURFBRU/HVWVhYffPCBzv8fK6IvcQiCIAiCULeZyB2AUPsEBARw/PhxWrRoodNxli5diouLS4mkTfjHunXruHz5MrNmzdLo+KVLl5b4OisriwULFgCUSKhqmr7EIQiCIAhC3SYSpzpKkiRycnKwtLTU+rnt7Ozo2rWr1s8r6JauE90i+fn5KBQKTEzEnx9BEIS6qLz3gaysLKysrKp8bl1e3wiCWKpn4P7880/atGmDubk5/v7+LFmyhA8++ACFQlHiOIVCwWuvvcby5ctp3rw55ubm/PzzzwAsWLCALl264OTkhJ2dHQEBAaxcuRJJkkqcIz8/n7fffht3d3esrKzo0aMHp06deiymspbqnTlzhuHDh+Pk5ISFhQXt27fn999/L3HMmjVrUCgUHDx4kFdffRUXFxecnZ0ZNWoU0dHRxcf5+vpy5coVDh06hEKhQKFQ4OvrW+HrFR0dzdixY7G1tcXe3p5x48YRGxtb6rGaxJuVlcWcOXPw8/PDwsICJycnOnbsyPr160scd/LkSYYNG4azszMWFhY0bNjwsZmg8PBwJkyYgKurK+bm5jRv3pzvv/++1Nd2/fr1zJs3D09PT+zs7OjXrx83btwoPi4oKIgdO3Zw79694tfn0Z+JR/17qd7du3epV68eoP75KHr8v2f3KhPv2rVr+c9//oOXlxfm5ubcunWLhIQEpk+fTosWLbCxscHV1ZU+ffoQGhpa/HhtxaFSqfjoo49o2rQplpaWODg40KZNG5YsWVLuazJt2jQsLCw4e/ZsiXP17dsXNzc3YmJiyn28IAhCbVfR3+Dy3gcmT56MjY0Nly5dYsCAAdja2tK3b18AkpOTmT59Ol5eXpiZmeHv78+8efPIzc0tMX551zeCoG3iI18DtmvXLkaNGkXPnj3ZuHEjBQUFfPnll8TFxZV6/NatWwkNDeX999/H3d0dV1dXQH1x+sorr1C/fn0ATpw4weuvv05UVBTvv/9+8eNfeuklfvnlF+bMmUP//v25fPkyo0aNIj09vcJYDx48yJNPPkmXLl1Yvnw59vb2bNiwgXHjxpGVlfXYcrsXX3yRIUOGsG7dOiIjI3nrrbd47rnnOHDgAABbtmxhzJgx2NvbFy8xMzc3LzeG7Oxs+vXrR3R0NIsWLaJJkybs2LGDcePGVTne2bNns3btWj766CPat29PZmYmly9fJikpqfhcu3fvZtiwYTRv3pyvv/6a+vXrc/fuXfbs2VN8zNWrV+nevTv169fnq6++wt3dnd27dzNz5kwSExOZP39+ifjee+89nnjiCVasWEFaWhrvvPMOw4YN49q1axgbG7N06VJefvllbt++zZYtWyr8/3mUh4cHu3bt4sknn2Tq1Km8+OKLAMVJTGXjnTt3Lt26dWP58uUYGRnh6upKQkICAPPnz8fd3Z2MjAy2bNlCUFAQ+/fvJygoSGtxfP7553zwwQf897//pWfPnuTn53P9+nVSUlLKfR0WL17MyZMnGTt2LGfPnsXBwYEFCxYQEhLCrl278PDwqPRrKwiCUFtU5r2gtPcBgLy8PIYPH84rr7zCu+++S0FBATk5OfTu3Zvbt2+zYMEC2rRpQ2hoKIsWLeL8+fPs2LGjRBxlXd8IgtZJgsHq1KmT5OPjI+Xm5hbfl56eLjk7O0uP/tcCkr29vZScnFzuOZVKpZSfny8tXLhQcnZ2llQqlSRJknTt2jUJkN58880Sx//2228SIE2aNKn4voMHD0qAdPDgweL7mjVrJrVv317Kz88v8fihQ4dKHh4eklKplCRJklavXi0B0vTp00sc9/nnn0uAFBMTU3xfy5YtpV69epX7fP5t2bJlEiD9+eefJe5/6aWXJEBavXp1peNt1aqV9NRTT5U7bsOGDaWGDRtK2dnZZR4zcOBAydvbW0pNTS1x/2uvvSZZWFgU/78VvbaDBw8ucdzvv/8uAdLx48eL7xsyZIjUoEGDcmP7t169epV4PRMSEiRAmj9/frXj7dmzZ4XjFxQUSPn5+VLfvn2lkSNHajWOoUOHSu3ataswhtKEh4dLdnZ20lNPPSXt27dPMjIykv773/9W6VyCIAi1iSZ/g8t7H5g0aZIESKtWrSpx//LlyyVA+v3330vc/9lnn0mAtGfPnuL7NL2+EQRtEEv1DFRmZiZnzpzhqaeewszMrPh+Gxsbhg0bVupj+vTpg6Oj42P3HzhwgH79+mFvb4+xsTGmpqa8//77JCUlER8fD6hnYACeffbZEo8dO3ZshbUqt27d4vr168WPLSgoKL4NHjyYmJiYEsvMAIYPH17i6zZt2gBw7969cscCUCqVJcZQqVTFz8HW1vaxc0+YMKHK8Xbu3Jm///6bd999l5CQELKzs0uc6+bNm9y+fZupU6diYWFRarw5OTns37+fkSNHYmVl9dh4OTk5nDhxQmuvT3VVJd7Ro0eXeq7ly5cTEBCAhYUFJiYmmJqasn//fq5du6bVODp37syFCxeYPn06u3fvJi0tTePn26hRI3766Se2bt3K0KFDCQwM5IMPPtD48YIgCLVRZd8LynofKO17Bw4cwNramjFjxpS4v2i1x/79+0vcX9b1jSBom0icDNTDhw+RJAk3N7fHvlfafUCpy4pOnTrFgAEDAPjpp584evQop0+fZt68eQDFiUDR0jN3d/cSjzcxMcHZ2bncWIuWDs6ZMwdTU9MSt+nTpwOQmJhY4jGPnrNoGd6jiUlpGjZsWGKMhQsXFj+H0l6bR59TZeL95ptveOedd9i6dSu9e/fGycmJp556ivDwcIDi5Wje3t5lxpuUlERBQQHffvvtY+MNHjxY669PdVUl3tJ+9r7++mteffVVunTpwubNmzlx4gSnT5/mySef1Oh5VCaOuXPn8uWXX3LixAkGDRqEs7Mzffv25cyZMxo95yFDhuDm5kZOTg6zZ8/G2NhYo8cJgiDUVpV9LyhrabOVlRV2dnaPndvd3f2x2lxXV1dMTExKLIcv79yCoG2ixslAOTo6olAoSq1nKqvZQWnNATZs2ICpqSnbt28vMSOydevWEscVXajHxsbi5eVVfH9BQcFjf8Ae5eLiAqgvXkeNGlXqMU2bNi33HJWxbdu2EsWjnp6egPo5lNbM4tHXqzLxWltbs2DBAhYsWEBcXFzx7NOwYcO4fv16cS3OgwcPyozX0dERY2Njnn/+eWbMmFHqMX5+fmU+vqZVJd7SfvZ+/fVXgoKCWLZsWYn7NamZq2wcJiYmzJ49m9mzZ5OSksK+fft47733GDhwIJGRkRV2cJo2bRrp6em0bNmSmTNnEhgYKD7dFAShTtP0b/ClS5eA0t8Hyrrf2dmZkydPIklSie/Hx8dTUFBQ/D5d3jkEQRdE4mSgrK2t6dixI1u3buXLL78sXq6XkZHB9u3bNT5PUTvQf3+Cnp2dzdq1a0scV9Rt7bfffqNDhw7F9//+++8UFBSUO0bTpk1p3LgxFy5c4JNPPtE4toqYm5uXOjPRunXrUo/v3bs3v//+O3/99VeJpW7r1q3TSrxubm5MnjyZCxcusHjxYrKysmjSpAkNGzZk1apVzJ49u9QGFlZWVvTu3ZuwsDDatGlTYulldZT1+lTm8fD4LJa24lUoFI+9HhcvXuT48eP4+PjoLA4HBwfGjBlDVFQUs2bN4u7du+W2Yl+xYgW//vorq1atolevXgQEBPDCCy889uGCIAhCXaKr9y6Avn378vvvv7N161ZGjhxZfP8vv/xS/H1BkINInAzYwoULGTJkCAMHDuSNN95AqVTyxRdfYGNjQ3JyskbnGDJkCF9//TUTJkzg5ZdfJikpiS+//PKxC9rmzZvz3HPPsXjxYkxNTenXrx+XL1/myy+/fGyKvTQ//PADgwYNYuDAgUyePBkvLy+Sk5O5du0a586dY9OmTZV+/q1bt2bDhg1s3LgRf39/LCwsykyaACZOnMj//d//MXHiRD7++GMaN27Mzp072b17d5Xj7dKlC0OHDqVNmzY4Ojpy7do11q5dS7du3YpnMb7//nuGDRtG165defPNN6lfvz73799n9+7d/PbbbwAsWbKEHj16EBgYyKuvvoqvry/p6encunWLbdu2FXcTrOzrExwczLJly+jQoQNGRkZ07NhR48fb2trSoEED/vzzT/r27YuTkxMuLi74+vpqJd6hQ4fy4YcfMn/+fHr16sWNGzdYuHAhfn5+JZJxbcQxbNgwWrVqRceOHalXrx737t1j8eLFNGjQgMaNG5cZ46VLl5g5cyaTJk3ihRdeAGDlypWMGTOGxYsXa7y5sCAIQm2ki/cuUL9ff//990yaNIm7d+/SunVrjhw5wieffMLgwYPp16+flp+JIGhI7u4UQvVs2bJFat26tWRmZibVr19f+vTTT6Line truncated
},
"type": "node"
}
],
"instanceArgs": {
"namespace": "html",
"tag": "div"
},
"nodeType": "DOM",
"props": {
"className": "interact-flex-row interact-widget"
},
"type": "node"
}
}
},
"systemjs_options": null
},
"nodeType": "Scope",
"props": {},
"type": "node"
}
],
"instanceArgs": {
"namespace": "html",
"tag": "div"
},
"nodeType": "DOM",
"props": {},
"type": "node"
},
"text/html": [
"<div\n",
" class=\"webio-mountpoint\"\n",
" data-webio-mountpoint=\"16925111089940656389\"\n",
">\n",
" <script>\n",
" (function(){\n",
" // Some integrations (namely, IJulia/Jupyter) use an alternate render pathway than\n",
" // just putting the html on the page. If WebIO isn't defined, then it's pretty likely\n",
" // that we're in one of those situations and the integration just isn't installed\n",
" // correctly.\n",
" if (typeof window.WebIO === \"undefined\") {\n",
" document\n",
" .querySelector('[data-webio-mountpoint=\"16925111089940656389\"]')\n",
" .innerHTML = (\n",
" '<div style=\"padding: 1em; background-color: #f8d6da; border: 1px solid #f5c6cb; font-weight: bold;\">' +\n",
" '<p><strong>WebIO not detected.</strong></p>' +\n",
" '<p>Please read ' +\n",
" '<a href=\"https://juliagizmos.github.io/WebIO.jl/latest/troubleshooting/not-detected/\" target=\"_blank\">the troubleshooting guide</a> ' +\n",
" 'for more information on how to resolve this issue.</p>' +\n",
" '<p><a href=\"https://juliagizmos.github.io/WebIO.jl/latest/troubleshooting/not-detected/\" target=\"_blank\">https://juliagizmos.github.io/WebIO.jl/latest/troubleshooting/not-detected/</a></p>' +\n",
" '</div>'\n",
" );\n",
" return;\n",
" }\n",
" WebIO.mount(\n",
" document.querySelector('[data-webio-mountpoint=\"16925111089940656389\"]'),\n",
" {\"props\":{},\"nodeType\":\"DOM\",\"type\":\"node\",\"instanceArgs\":{\"namespace\":\"html\",\"tag\":\"div\"},\"children\":[{\"props\":{\"className\":\"field interact-widget\"},\"nodeType\":\"DOM\",\"type\":\"node\",\"instanceArgs\":{\"namespace\":\"html\",\"tag\":\"div\"},\"children\":[{\"props\":{},\"nodeType\":\"Scope\",\"type\":\"node\",\"instanceArgs\":{\"imports\":{\"data\":[{\"name\":\"knockout\",\"type\":\"js\",\"url\":\"\\/assetserver\\/2df50fa7f121fedbb109a97bba894e56aef8c852-knockout.js\"},{\"name\":\"knockout_punches\",\"type\":\"js\",\"url\":\"\\/assetserver\\/bb2c111220e03062c2973eb306d97c8fc96857d6-knockout_punches.js\"},{\"name\":null,\"type\":\"js\",\"url\":\"\\/assetserver\\/dc651e248070c73d6a4eaa2766193d4a09a13f05-all.js\"},{\"name\":null,\"type\":\"css\",\"url\":\"\\/assetserver\\/0318b018d0ff756043bf1ee25f56ab5c74ef7889-style.css\"},{\"name\":null,\"type\":\"css\",\"url\":\"\\/assetserver\\/f798281fed8908611ae453136bda5c1a2e6bffd7-bulma_confined.min.css\"}],\"type\":\"async_block\"},\"id\":\"7102712321243811129\",\"handlers\":{\"changes\":[\"(function (val){return (val!=this.model[\\\"changes\\\"]()) ? (this.valueFromJulia[\\\"changes\\\"]=true, this.model[\\\"changes\\\"](val)) : undefined})\"],\"index\":[\"(function (val){return (val!=this.model[\\\"index\\\"]()) ? (this.valueFromJulia[\\\"index\\\"]=true, this.model[\\\"index\\\"](val)) : undefined})\"]},\"systemjs_options\":null,\"mount_callbacks\":[\"function () {\\n var handler = (function (ko, koPunches) {\\n ko.punches.enableAll();\\n ko.bindingHandlers.numericValue = {\\n init: function(element, valueAccessor, allBindings, data, context) {\\n var stringified = ko.observable(ko.unwrap(valueAccessor()));\\n stringified.subscribe(function(value) {\\n var val = parseFloat(value);\\n if (!isNaN(val)) {\\n valueAccessor()(val);\\n }\\n });\\n valueAccessor().subscribe(function(value) {\\n var str = JSON.stringify(value);\\n if ((str == \\\"0\\\") && ([\\\"-0\\\", \\\"-0.\\\"].indexOf(stringified()) >= 0))\\n return;\\n if ([\\\"null\\\", \\\"\\\"].indexOf(str) >= 0)\\n return;\\n stringified(str);\\n });\\n ko.applyBindingsToNode(\\n element,\\n {\\n value: stringified,\\n valueUpdate: allBindings.get('valueUpdate'),\\n },\\n context,\\n );\\n }\\n };\\n var json_data = {\\\"formatted_vals\\\":[\\\"0.05\\\",\\\"0.1\\\",\\\"0.15\\\",\\\"0.2\\\",\\\"0.25\\\",\\\"0.3\\\",\\\"0.35\\\",\\\"0.4\\\",\\\"0.45\\\",\\\"0.5\\\",\\\"0.55\\\",\\\"0.6\\\",\\\"0.65\\\",\\\"0.7\\\",\\\"0.75\\\",\\\"0.8\\\",\\\"0.85\\\",\\\"0.9\\\",\\\"0.95\\\",\\\"1.0\\\",\\\"1.05\\\",\\\"1.1\\\",\\\"1.15\\\",\\\"1.2\\\",\\\"1.25\\\",\\\"1.3\\\",\\\"1.35\\\",\\\"1.4\\\",\\\"1.45\\\",\\\"1.5\\\",\\\"1.55\\\",\\\"1.6\\\",\\\"1.65\\\",\\\"1.7\\\",\\\"1.75\\\",\\\"1.8\\\",\\\"1.85\\\",\\\"1.9\\\",\\\"1.95\\\",\\\"2.0\\\"],\\\"changes\\\":WebIO.getval({\\\"name\\\":\\\"changes\\\",\\\"scope\\\":\\\"7102712321243811129\\\",\\\"id\\\":\\\"1344084652692618713\\\",\\\"type\\\":\\\"observable\\\"}),\\\"index\\\":WebIO.getval({\\\"name\\\":\\\"index\\\",\\\"scope\\\":\\\"7102712321243811129\\\",\\\"id\\\":\\\"10488540316168852659\\\",\\\"type\\\":\\\"observable\\\"})};\\n var self = this;\\n function AppViewModel() {\\n for (var key in json_data) {\\n var el = json_data[key];\\n this[key] = Array.isArray(el) ? ko.observableArray(el) : ko.observable(el);\\n }\\n \\n [this[\\\"formatted_val\\\"]=ko.computed( function(){\\n return this.formatted_vals()[parseInt(this.index())-(1)];\\n }\\n,this)]\\n [this[\\\"changes\\\"].subscribe((function (val){!(this.valueFromJulia[\\\"changes\\\"]) ? (WebIO.setval({\\\"name\\\":\\\"changes\\\",\\\"scope\\\":\\\"7102712321243811129\\\",\\\"id\\\":\\\"1344084652692618713\\\",\\\"type\\\":\\\"observable\\\"},val)) : undefined; return this.valueFromJulia[\\\"changes\\\"]=false}),self),this[\\\"index\\\"].subscribe((function (val){!(this.valueFromJulia[\\\"index\\\"]) ? (WebIO.setval({\\\"name\\\":\\\"index\\\",\\\"scope\\\":\\\"7102712321243811129\\\",\\\"id\\\":\\\"10488540316168852659\\\",\\\"type\\\":\\\"observable\\\"},val)) : undefined; return this.valueFromJulia[\\\"index\\\"]=false}),self)]\\n \\n }\\n self.model = new AppViewModel();\\n self.valueFromJulia = {};\\n for (var key in json_data) {\\n self.valueFromJulia[key] = false;\\n }\\n ko.applyBindings(self.model, self.dom);\\n}\\n);\\n (WebIO.importBlock({\\\"data\\\":[{\\\"name\\\":\\\"knockout\\\",\\\"type\\\":\\\"js\\\",\\\"url\\\":\\\"\\/assetserver\\/2df50fa7f121fedbb109a97bba894e56aef8c852-kLine truncated
" window,\n",
" );\n",
" })()\n",
" </script>\n",
"</div>\n"
],
"text/plain": [
"Node{WebIO.DOM}(WebIO.DOM(:html, :div), Any[Node{WebIO.DOM}(WebIO.DOM(:html, :div), Any[Scope(Node{WebIO.DOM}(WebIO.DOM(:html, :div), Any[Node{WebIO.DOM}(WebIO.DOM(:html, :div), Any[Node{WebIO.DOM}(WebIO.DOM(:html, :label), Any[\"s × L\"], Dict{Symbol, Any}(:className => \"interact \", :style => Dict{Any, Any}(:padding => \"5px 10px 0px 10px\")))], Dict{Symbol, Any}(:className => \"interact-flex-row-left\")), Node{WebIO.DOM}(WebIO.DOM(:html, :div), Any[Node{WebIO.DOM}(WebIO.DOM(:html, :input), Any[], Dict{Symbol, Any}(:max => 40, :min => 1, :attributes => Dict{Any, Any}(:type => \"range\", Symbol(\"data-bind\") => \"numericValue: index, valueUpdate: 'input', event: {change: function (){this.changes(this.changes()+1)}}\", \"orient\" => \"horizontal\"), :step => 1, :className => \"slider slider is-fullwidth\", :style => Dict{Any, Any}()))], Dict{Symbol, Any}(:className => \"interact-flex-row-center\")), Node{WebIO.DOM}(WebIO.DOM(:html, :div), Any[Node{WebIO.DOM}(WebIO.DOM(:html, :p), Any[], Dict{Symbol, Any}(:attributes => Dict(\"data-bind\" => \"text: formatted_val\")))], Dict{Symbol, Any}(:className => \"interact-flex-row-right\"))], Dict{Symbol, Any}(:className => \"interact-flex-row interact-widget\")), Dict{String, Tuple{AbstractObservable, Union{Nothing, Bool}}}(\"changes\" => (Observable(0), nothing), \"index\" => (Observable{Any}(6), nothing)), Set{String}(), nothing, Asset[Asset(\"js\", \"knockout\", \"/Users/stevenj/.julia/packages/Knockout/HReiN/src/../assets/knockout.js\"), Asset(\"js\", \"knockout_punches\", \"/Users/stevenj/.julia/packages/Knockout/HReiN/src/../assets/knockout_punches.js\"), Asset(\"js\", nothing, \"/Users/stevenj/.julia/packages/InteractBase/Qhvxg/src/../assets/all.js\"), Asset(\"css\", nothing, \"/Users/stevenj/.julia/packages/InteractBase/Qhvxg/src/../assets/style.css\"), Asset(\"css\", nothing, \"/Users/stevenj/.julia/packages/Interact/PENUy/src/../assets/bulma_confined.min.css\")], Dict{Any, Any}(\"changes\" => Any[WebIO.JSString(\"(function (val){return (val!=this.model[\\\"changes\\\"]()) ? (this.valueFromJulia[\\\"changes\\\"]=true, this.model[\\\"changes\\\"](val)) : undefined})\")], \"index\" => Any[WebIO.JSString(\"(function (val){return (val!=this.model[\\\"index\\\"]()) ? (this.valueFromJulia[\\\"index\\\"]=true, this.model[\\\"index\\\"](val)) : undefined})\")]), WebIO.ConnectionPool(Channel{Any}(32), Set{AbstractConnection}(), Condition(Base.InvasiveLinkedList{Task}(Task (runnable) @0x0000000112f7fb20, Task (runnable) @0x0000000112f7fb20), Base.AlwaysLockedST(1))), WebIO.JSString[WebIO.JSString(\"function () {\\n var handler = (function (ko, koPunches) {\\n ko.punches.enableAll();\\n ko.bindingHandlers.numericValue = {\\n init: function(element, valueAccessor, allBindings, data, context) {\\n var stringified = ko.observable(ko.unwrap(valueAccessor()));\\n stringified.subscribe(function(value) {\\n var val = parseFloat(value);\\n if (!isNaN(val)) {\\n valueAccessor()(val);\\n }\\n });\\n valueAccessor().subscribe(function(value) {\\n var str = JSON.stringify(value);\\n if ((str == \\\"0\\\") && ([\\\"-0\\\", \\\"-0.\\\"].indexOf(stringified()) >= 0))\\n return;\\n if ([\\\"null\\\", \\\"\\\"].indexOf(str) >= 0)\\n return;\\n stringified(str);\\n });\\n ko.applyBindingsToNode(\\n element,\\n {\\n value: stringified,\\n valueUpdate: allBindings.get('valueUpdate'),\\n },\\n context,\\n );\\n }\\n };\\n var json_data = {\\\"formatted_vals\\\":[\\\"0.05\\\",\\\"0.1\\\",\\\"0.15\\\",\\\"0.2\\\",\\\"0.25\\\",\\\"0.3\\\",\\\"0.35\\\",\\\"0.4\\\",\\\"0.45\\\",\\\"0.5\\\",\\\"0.55\\\",\\\"0.6\\\",\\\"0.65\\\",\\\"0.7\\\",\\\"0.75\\\",\\\"0.8\\\",\\\"0.85\\\",\\\"0.9\\\",\\\"0.95\\\",\\\"1.0\\\",\\\"1.05\\\",\\\"1.1\\\",\\\"1.15\\\",\\\"1.2\\\",\\\"1.25\\\",\\\"1.3\\\",\\\"1.35\\\",\\\"1.4\\\",\\\"1.45\\\",\\\"1.5\\\",\\\"1.55\\\",\\\"1.6\\\",\\\"1.65\\\",\\\"1.7\\\",\\\"1.75\\\",\\\"1.8\\\",\\\"1.85\\\",\\\"1.9\\\",\\\"1.95\\\",\\\"2.0\\\"],\\\"changes\\\":WebIO.getval({\\\"name\\\":\\\"changes\\\",\\\"scope\\\":\\\"7102712321243811129\\\",\\\"id\\\":\\\"1344084652692618713\\\",\\\"type\\\":\\\"observable\\\"}),\\\"index\\\":WebIO.getval({\\\"name\\\":\\\"index\\\",\\\"scope\\\":\\\"7102712321243811129\\\",\\\"id\\\":\\\"10488540316168852659\\\",\\\"type\\\":\\\"observable\\\"})};\\n var self = this;\\n function AppViewModel() {\\n for (var key in json_data) {\\n var el = json_data[key];\\n this[key] = Array.isArray(el) ? ko.observableArray(el) : ko.observable(el);\\n }\\n \\n [this[\\\"formatted_val\\\"]=ko.computed( function(){\\n returLine truncated
]
},
"execution_count": 81,
"metadata": {
"application/vnd.webio.node+json": {
"kernelId": "aa0ee44b-aa39-4984-abc8-16acd87b23dd"
}
},
"output_type": "execute_result"
}
],
"source": [
"using Interact\n",
"\n",
"f(x) = norm(b-A*x)^2 / m\n",
"f(x1,x2) = f([x1,x2])\n",
"x1 = range(-2,3, length=100)\n",
"x2 = range(0,4,length=100)\n",
"fvals = f.(x1',x2)\n",
"\n",
"@show L = (2/m) * maximum(eigvals(A'*A)) # Lipschitz constant\n",
"\n",
"f1 = figure(figsize=(10,4))\n",
"@manipulate for sL in slider(0.05:0.05:2, value=0.3, label=\"s × L\"),\n",
" iterations in slider(10:100, value=20, label=\"# iterations\")\n",
" withfig(f1) do\n",
" s = sL / L\n",
" x = [0.0,0.0]\n",
" X = [x]\n",
" ΔX = [norm(x - x̂)]\n",
" for i = 1:iterations\n",
" ∇f = (2/m)*(A'*(A*x - b))\n",
" x = x - s * ∇f\n",
" push!(X, x)\n",
" push!(ΔX, norm(x - x̂))\n",
" end\n",
" \n",
" subplot(1,2,1)\n",
" plt.subplots_adjust(wspace=0.4)\n",
" contour(x1', x2, fvals)\n",
" plot(x̂[1], x̂[2], \"r*\")\n",
" plot(getindex.(X,1), getindex.(X,2), \"ko-\")\n",
" xlabel(L\"x_1\")\n",
" ylabel(L\"x_2\")\n",
" title(\"gradient-descent iterates x\")\n",
" \n",
" subplot(1,2,2)\n",
" semilogy(ΔX, \"ko-\")\n",
" ylim(1e-10, 2)\n",
" xlabel(\"iteration\")\n",
" ylabel(L\"\\Vert x - \\hat{x} \\Vert\")\n",
" title(\"error\")\n",
" end\n",
"end"
]
},
{
"cell_type": "markdown",
"id": "8b4d9fa9",
"metadata": {},
"source": [
"## Stochastic gradient descent\n",
"\n",
"In the most basic form of stochastic gradient descent, we look at one data point at a time, chosen at random. More generally, we can look at \"mini-batches\" of $M$ data points at a time. That is, we will only **estimate**\n",
"$$\n",
"\\nabla f = E[\\nabla f_i] \\approx \\frac{1}{M} \\sum_{i \\in \\text{mini-batch}} \\nabla f_i\n",
"$$\n",
"The simplest case is $M=1$: we only look at one data point at a time!\n",
"\n",
"This is *much* cheaper than computing $\\nabla f$ when the size of the training is large. (In fact, we can even handle *infinite* training sets this way, where the training data is generated randomly as needed.)\n",
"\n",
"Let's see how it works for the problem above, sampling data at random from an infinite ($m=\\infty$) training set generated as needed from the same model as above:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cc7d34e5",
"metadata": {},
"outputs": [],
"source": [
"using Interact\n",
"\n",
"x̂₀ = [1,2] # ground-truth model\n",
"\n",
"f2 = figure(figsize=(10,4))\n",
"@manipulate for sL in slider(0.05:0.05:2, value=0.3, label=\"s × L\"),\n",
" M in slider(1:10, value=1, label=\"mini-batch M\")\n",
" iterations in slider(10:100, value=20, label=\"# iterations\")\n",
" withfig(f2) do\n",
" s = sL / L\n",
" x = [0.0,0.0]\n",
" X = [x]\n",
" ΔX = [norm(x - x̂₀)]\n",
" for i = 1:iterations\n",
" A\\_M\n",
" ∇f = (2/m)*(A'*(A*x - b))\n",
" x = x - s * ∇f\n",
" push!(X, x)\n",
" push!(ΔX, norm(x - x̂))\n",
" end\n",
" \n",
" subplot(1,2,1)\n",
" plt.subplots_adjust(wspace=0.4)\n",
" contour(x1', x2, fvals)\n",
" plot(x̂[1], x̂[2], \"r*\")\n",
" plot(getindex.(X,1), getindex.(X,2), \"ko-\")\n",
" xlabel(L\"x_1\")\n",
" ylabel(L\"x_2\")\n",
" title(\"gradient-descent iterates x\")\n",
" \n",
" subplot(1,2,2)\n",
" semilogy(ΔX, \"ko-\")\n",
" ylim(1e-10, 2)\n",
" xlabel(\"iteration\")\n",
" ylabel(L\"\\Vert x - \\hat{x} \\Vert\")\n",
" title(\"error\")\n",
" end\n",
"end"
]
}
],
"metadata": {
"@webio": {
"lastCommId": "1adf1e5916bf47bb8c202148ab48014f",
"lastKernelId": "aa0ee44b-aa39-4984-abc8-16acd87b23dd"
},
"kernelspec": {
"display_name": "Julia 1.8.0",
"language": "julia",
"name": "julia-1.8"
},
"language_info": {
"file_extension": ".jl",
"mimetype": "application/julia",
"name": "julia",
"version": "1.8.2"
}
},
"nbformat": 4,
"nbformat_minor": 5
}