daviddoji/Kalman-and-Bayesian-Filters-in-Python
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Made LaTeX font smaller.

Browse Source 
It was rather too large, especially when inline with text. I might
have gone just a *tad* too far in the other direction, but it mostly
looks very good now.
This commit is contained in:
Roger Labbe 2015-04-08 21:08:55 -07:00
parent 73f54b1970
commit 7f3a50aee4
18 changed files with 837 additions and 137588 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

55
00_Preface.ipynb
View File

@ -73,21 +73,21 @@
" }\n",
" .text_cell_render h2 {\n",
" font-weight: 200;\n",
" font-size: 20pt;\n",
" font-size: 16pt;\n",
" font-style: italic;\n",
" line-height: 100%;\n",
" color:#c76c0c;\n",
" margin-bottom: 0.5em;\n",
" margin-top: 1.5em;\n",
" display: block;\n",
" white-space: nowrap;\n",
" display: inline;\n",
" white-space: wrap;\n",
" } \n",
" h3 {\n",
" font-family: 'Open sans',verdana,arial,sans-serif;\n",
" }\n",
" .text_cell_render h3 {\n",
" font-weight: 300;\n",
" font-size: 18pt;\n",
" font-weight: 200;\n",
" font-size: 14pt;\n",
" line-height: 100%;\n",
" color:#d77c0c;\n",
" margin-bottom: 0.5em;\n",
@ -99,8 +99,8 @@
" font-family: 'Open sans',verdana,arial,sans-serif;\n",
" }\n",
" .text_cell_render h4 {\n",
" font-weight: 300;\n",
" font-size: 16pt;\n",
" font-weight: 100;\n",
" font-size: 14pt;\n",
" color:#d77c0c;\n",
" margin-bottom: 0.5em;\n",
" margin-top: 0.5em;\n",
@ -111,7 +111,7 @@
" font-family: 'Open sans',verdana,arial,sans-serif;\n",
" }\n",
" .text_cell_render h5 {\n",
" font-weight: 300;\n",
" font-weight: 200;\n",
" font-style: normal;\n",
" color: #1d3b84;\n",
" font-size: 16pt;\n",
@ -122,9 +122,9 @@
" }\n",
" div.text_cell_render{\n",
" font-family: 'Arimo',verdana,arial,sans-serif;\n",
" line-height: 135%;\n",
" font-size: 125%;\n",
" width:750px;\n",
" line-height: 125%;\n",
" font-size: 120%;\n",
" width:740px;\n",
" margin-left:auto;\n",
" margin-right:auto;\n",
" text-align:justify;\n",
@ -242,6 +242,7 @@
" },\n",
" displayAlign: 'center', // Change this to 'center' to center equations.\n",
" \"HTML-CSS\": {\n",
" scale:85,\n",
" styles: {'.MathJax_Display': {\"margin\": 4}}\n",
" }\n",
" });\n",
@ -274,7 +275,19 @@
{
"cell_type": "markdown",
"metadata": {},
"source": []
"source": [
"## Kalman and Bayesian Filters\n",
"\n",
"Sensors are noisy. The world is full of data and events that we want to measure and track, but we cannot rely on our sensors to give us perfect information. I have a digital scale at home; if I weigh the same object multiple times I often get slightly different readings. \n",
"\n",
"In simple cases the solution is obvious. If my scale gives slightly different readings I can just take a few readings and average them. But what do we do when the sensor is very noisy, or the information we are trying is very important? We may be trying to track the movement of an airplane. We may want to create an autopilot for a drone, track the milk production of cows, or ensure that our farm tractor seeded the entire field. I do computer vision, and I need to track moving objects in images. \n",
"\n",
"This book teaches you how to solve these sorts of filtering problems. We use several different algorithms, but they are all based on Bayesian probability. Later in the book Bayesian probability will be introduced with mathematical rigor, but in simple terms it is merely determining what is likely to be true based on past information. For example, suppose you drive home late one night and you see somebody with a mask on trying to open the window of your neighbors house. Absent any other information you might conclude that a break in is in progress. But you know what your neighbor looks like, and despite the mask this person looks like your neighbor. You are not sure, but your certainty that this is a crime lessens. Now suppose that you remember that this is October 31 - Halloween in the US, when we dress up in costumes. This provides a reason for the mask, and you form the hypothesis that your neighbor is locked outside of his house. Finally, suppose your wife pipes up with \"didn't they sell that house last week? Why is he trying to get inside\". This new piece of information seems to favor the crime hypothesis. \n",
"\n",
"There is more to Bayesian probability than that, but you have the main idea. Knowledge is uncertain, and we alter our beliefs based on *prior* evidence. How does this apply to filtering? Say we are trying to track an aircraft that up to now has been flying straight and level. Its on board navigation reports that it suddenly changed direction. Did it really do so, or is the sensor reporting bad data? It depends. If this is a jet fighter we may be very inclined to believe the report of a sudden maneuver. If it is a large, cumbersome passenger plane we might be very skeptical. Now say that in the next second the sensor reports a new position that is in line with the previous straight and level path. This leads us to make us think that the previous measurement was an error, though of course it could be true that it is *this* measurement that is wrong, not the last one.\n",
"\n",
"I've been using phrases like \"more likely.\" Bayesian filters use mathematics to compute these probabilities exactly. Kalman filters are a *type* of Bayesian filter. In other words, all Kalman filters are Bayesian filters, but not all Bayesian filters are Kalman filters. We will learn much more about how these two are related; for now it is enough to say that the Kalman filter can guarantee that its estimate is optimal under certain conditions, whereas other types of Bayesian filters are not necessarily optimal. "
]
},
{
"cell_type": "markdown",
@ -284,17 +297,13 @@
"source": [
"## Motivation\n",
"\n",
"Sensors are noisy. The world is full of data and events that we want to measure and track, but we cannot rely on our sensors to give us perfect information. I have a digital scale at home; if I weigh the same object multiple times I often get slightly different readings. \n",
"\n",
"In simple cases the solution is obvious. If my scale gives slightly different readings I can just take a few readings and average them. But what do we do when the sensor is very noisy, or the information we are trying is very important? We may be trying to track the movement of an airplane. We may want to create an autopilot for a drone, track the milk production of cows, or ensure that our farm tractor seeded the entire field. I do computer vision, and I need to track moving objects in images. \n",
"\n",
"The Kalman filter was invented by Rudolf Emil Kálmán to solve this sort of problem in a mathematically optimal way. The theory and applications are beautiful, but quite difficult to learn if you are not already well trained in topics such as signal processing, control theory, probability and statistics, and guidance and control theory. Its first use was on the Apollo missions to the moon, and since then it has been used in an enormous variety of domains. There are Kalman filters in aircraft, on submarines, on cruise missles. Wall street uses them to track the market. They are used in robots, in IoT (Internet of Things) sensors, and in laboratory instruments. Chemical plants use them to control and monitor reactions. They are used in medical equipment to do things like medical imaging or to remove noise from cardiac signals. If it involves a sensor, a Kalman filter or a close relative to the Kalman filter is usually involved.\n",
"The Kalman filter was invented by Rudolf Emil Kálmán to solve this sort of problem in a mathematically optimal way. The theory and applications are beautiful, but quite difficult to learn if you are not already well trained in topics such as signal processing, control theory, probability and statistics, and guidance and control theory. Its first use was on the Apollo missions to the moon, and since then it has been used in an enormous variety of domains. There are Kalman filters in aircraft, on submarines, on cruise missiles. Wall street uses them to track the market. They are used in robots, in IoT (Internet of Things) sensors, and in laboratory instruments. Chemical plants use them to control and monitor reactions. They are used in medical equipment to do things like medical imaging or to remove noise from cardiac signals. If it involves a sensor, a Kalman filter or a close relative to the Kalman filter is usually involved.\n",
"\n",
"I'm a software engineer that spent almost two decades in aerospace, and so I have always been 'bumping elbows' with the Kalman filter, but never implemented one. They always had a fearsome reputation for difficulty, and I did not have the requisite education. Everyone I met that did implement them had multiple graduate courses on the topic and extensive industrial experience with them. As I moved into solving tracking problems with computer vision the need to implement them myself became urgent. There are classic textbooks in the field, such as Grewal and Andrew's excellent *Kalman Filtering*. But sitting down and trying to read many of these books is a dismal and trying experience if you do not have the background. Typically the first few chapters fly through several years of undergraduate math, blithely referring you to textbooks on, for example, Itō calculus, and presenting an entire semester's worth of statistics in a few brief paragraphs. These books are good textbooks for an upper undergraduate course, and an invaluable reference to researchers and professionals, but the going is truly difficult for the more casual reader. Symbology is introduced without explanation, different texts use different words and variables names for the same concept, and the books are almost devoid of examples or worked problems. I often found myself able to parse the words and comprehend the mathematics of a definition, but had no idea as to what real world phenomena these words and math were attempting to describe. \"But what does that *mean?*\" was my repeated thought.\n",
"\n",
"However, as I began to finally understand the Kalman filter I realized the underlying concepts are quite straightforward. A few simple probability rules, some intuition about how we integrate disparate knowledge to explain events in our everyday life and the core concepts of the Kalman filter are accessible. Kalman filters have a reputation for difficulty, but shorn of much of the formal terminology the beauty of the subject and of their math became clear to me, and I fell in love with the topic. \n",
"\n",
"As I began to understand the math and theory more difficulties appeard. A book or paper will make some statement of fact and presents a graph as proof. Unfortunately, why the statement is true is not clear to me, nor is the method by which you might make that plot obvious. Or maybe I wonder \"is this true if R=0?\" Or the author provides pseudocode - at such a high level that the implementation is not obvious. Some books offer Matlab code, but I do not have a license to that expensive package. Finally, many books end each chapter with many useful exercises. Exercises which you need to understand if you want to implement Kalman filters for yourself, but exercises with no answers. If you are using the book in a classroom, perhaps this is okay, but it is terrible for the independent reader. I loathe that an author withholds information from me, presumably to avoid 'cheating' by the student in the classroom.\n",
"As I began to understand the math and theory more difficulties appeared. A book or paper will make some statement of fact and presents a graph as proof. Unfortunately, why the statement is true is not clear to me, nor is the method by which you might make that plot obvious. Or maybe I wonder \"is this true if R=0?\" Or the author provides pseudocode - at such a high level that the implementation is not obvious. Some books offer Matlab code, but I do not have a license to that expensive package. Finally, many books end each chapter with many useful exercises. Exercises which you need to understand if you want to implement Kalman filters for yourself, but exercises with no answers. If you are using the book in a classroom, perhaps this is okay, but it is terrible for the independent reader. I loathe that an author withholds information from me, presumably to avoid 'cheating' by the student in the classroom.\n",
"\n",
"None of this necessary, from my point of view. Certainly if you are designing a Kalman filter for a aircraft or missile you must thoroughly master of all of the mathematics and topics in a typical Kalman filter textbook. I just want to track an image on a screen, or write some code for my Arduino project. I want to know how the plots in the book are made, and chose different parameters than the author chose. I want to run simulations. I want to inject more noise in the signal and see how a filter performs. There are thousands of opportunities for using Kalman filters in everyday code, and yet this fairly straightforward topic is the provenance of rocket scientists, medical engineers and academics.\n",
"\n",
@ -306,7 +315,7 @@
"\n",
"This book has supporting libraries for computing statistics, plotting various things related to filters, and for the various filters that we cover. This does require a strong caveat; most of the code is written for didactic purposes. It is rare that I chose the most efficient solution (which often obscures the intent of the code), and in the first parts of the book I did not concern myself with numerical stability. This is important to understand - Kalman filters in aircraft are carefully designed and implemented to be numerically stable; the naive implementation is not stable in many cases. If you are serious about Kalman filters this book will not be the last book you need. My intention is to introduce you to the concepts and mathematics, and to get you to the point where the textbooks are approachable.\n",
"\n",
"Finally, this book is free. The cost for the books required to learn Kalman filtering is somewhat prohibitive even for a Silicon Valley engineer like myself; I cannot believe the are within the reach of someone in a depressed economy, or a financially struggling student. I have gained so much from free software like Python, and free books like those from Allen B. Downey [here](http://www.greenteapress.com/) [1]. It's time to repay that. So, the book is free, it is hosted on free servers, and it uses only free and open software such as IPython and mathjax to create the book. "
"Finally, this book is free. The cost for the books required to learn Kalman filtering is somewhat prohibitive even for a Silicon Valley engineer like myself; I cannot believe the are within the reach of someone in a depressed economy, or a financially struggling student. I have gained so much from free software like Python, and free books like those from Allen B. Downey [here](http://www.greenteapress.com/) [1]. It's time to repay that. So, the book is free, it is hosted on free servers, and it uses only free and open software such as IPython and MathJax to create the book. "
]
},
{
@ -391,7 +400,7 @@
"\n",
"I do not cover how to install IPython in this book; requirements change based on what other python installations you may have, whether you use a third party package like Anaconda Python, what operating system you are using, and so on.\n",
"\n",
"The IPython Notebook format was changed as of IPython 3.0. If you are running 2.4 you will still be able to open and run the notebooks, but they will be downconverted for you. If you make changes DO NOT push 2.4 version notebooks to me! I strongly recommend updating to 3.0 as soon as possible, as this format change will just become more frustrating with time.\n",
"The IPython Notebook format was changed as of IPython 3.0. If you are running 2.4 you will still be able to open and run the notebooks, but they will be down converted for you. If you make changes DO NOT push 2.4 version notebooks to me! I strongly recommend updating to 3.0 as soon as possible, as this format change will just become more frustrating with time.\n",
"\n",
"You will need Python 2.7 or later installed. Almost all of my work is done in Python 3.4, but I periodically test on 2.7. I do not promise any specific check in will work in 2.7 however. I do use Python's \"from __future__ import ...\" statement to help with compatibility. For example, all prints need to use parenthesis. If you try to add, say, \"print 3.14\" into the book your script will fail; you must write \"print (3.4)\" as in Python 3.X.\n",
"\n",
@ -451,15 +460,15 @@
"source": [
"I am first and foremost a programmer. Most Kalman filtering and other engineering texts are written by mathematicians or engineers. As a result, the software is usually not production quality. I will take Paul Zarchan's book *Fundamentals of Kalman Filtering* as an example. This is a fantastic book and it belongs in your library. But the code is Fortran listing without any subroutines beyond calls to functions like `MATMUL`. This means that Kalman filters are re-implemented over and over again throughout the book. The same listing mixes simulation code with filtering code, so until you become aware of the author's naming style it can be difficult to ascertain what code is the filter and what code is the simulation. Some chapters implement the same filter in subtly different ways, and uses bold text to highlight the few lines that changed. If he needs to use Runge Kutta, that is just embedded in the code, without comments. \n",
"\n",
"There's a better way. If I want to perform an SVD I call `svd`, I do not embed an SVD implementation in my code. This buys me several things. First, I don't have to re-implement SVD multiple times. I don't have to debug SVD several times, and if I do find a bug, I can fix it once and be assured that it now works across all my different projects. Finally, it is readable. It is rare that I care about the implementation of SVD in my projects.\n",
"There's a better way. If I want to perform an SVD I call `svd()`, I do not embed an SVD implementation in my code. This buys me several things. First, I don't have to re-implement SVD multiple times. I don't have to debug SVD several times, and if I do find a bug, I can fix it once and be assured that it now works across all my different projects. Finally, it is readable. It is rare that I care about the implementation of SVD in my projects.\n",
"\n",
"Now, this is a textbook on Kalman filtering, and you could reasonably point out that we *do* care about the implementation of Kalman filters. To an extent that is true, but as you will find out the code that performs the filtering amounts to 7 or so lines of code. The code to implement the math is fairly trivial. Most of the work that Kalman filters requires is the design of the matrices that get fed into the math engine. So that is how I have structured the code.\n",
"\n",
"For example, there is a class named `KalmanFilter` which implements the linear algebra for performing kalman filtering. To use it you will construct an object of that class, initialize various parameters, then enter a while loop where you call `KalmanFilter.predict()` and `KalmanFilter.update()` to incorporate your measurements. Thus most of your programs will be only 20-50 lines, most of that boilerplate - setting up the matrices, and then plotting and/or using the results. The possible downside of that is that the equations that perform the filtering are hidden behind functions, which we could argue is a loss in a pedagogical text. I argue the converse. I want you to learn how to use Kalman filters in the real world, for real projects, and you shouldn't be cutting and pasting established algorithms all over the place. If you want to use ode45 (Runga Kutta) you call that function, you don't re-implement it from scratch. We will do the same here.\n",
"For example, there is a class named `KalmanFilter` which implements the linear algebra for performing Kalman filtering. To use it you will construct an object of that class, initialize various parameters, then enter a while loop where you call `KalmanFilter.predict()` and `KalmanFilter.update()` to incorporate your measurements. Thus most of your programs will be only 20-50 lines, most of that boilerplate - setting up the matrices, and then plotting and/or using the results. The possible downside of that is that the equations that perform the filtering are hidden behind functions, which we could argue is a loss in a pedagogical text. I argue the converse. I want you to learn how to use Kalman filters in the real world, for real projects, and you shouldn't be cutting and pasting established algorithms all over the place. If you want to use ode45 (Runga Kutta) you call that function, you don't re-implement it from scratch. We will do the same here.\n",
"\n",
"However, it is undeniable that you will be a bit less versed with the equations of the Kalman filter as a result. I strongly recommend looking at the source for FilterPy, and maybe even implementing your own filter from scratch to be sure you understand the concepts.\n",
"\n",
"I use a fair number of classes in FilterPy. I do not use inheritance or virtual functions or any of that sort of OO design. I use classes as a way to organize the data that the filters require. For example, the `KalmanFilter` class mentioned above stores matrices called `x`, `P`, `R`, `Q`, and more. I've seen procedural libraries for Kalman filters, and they require the programmer to maintain all of those matrices. This perhaps isn't so bad for a toy program, but start programming, say, a bank of Kalman filters and you will not enjoy having to manage all of those matrices and other associated data.\n",
"I use a fair number of classes in FilterPy. I do not use inheritance or virtual functions or any of that sort of object oriented design. I use classes as a way to organize the data that the filters require. For example, the `KalmanFilter` class mentioned above stores matrices called `x`, `P`, `R`, `Q`, and more. I've seen procedural libraries for Kalman filters, and they require the programmer to maintain all of those matrices. This perhaps isn't so bad for a toy program, but start programming, say, a bank of Kalman filters and you will not enjoy having to manage all of those matrices and other associated data.\n",
"\n",
"A word on variable names. I am an advocate for descriptive variable names. `R` is not, normally, descriptive. `R` is the measurement noise covariance matrix, so I could reasonably call it `measurement_noise_covariance`, and I've seen libraries do that. I've chosen not to. Why? In the end, Kalman filtering is math. To write a Kalman filter you are going to have to start by sitting down with a piece of paper and doing some math. You will be writing normal algebraic equations. Also, every Kalman filter text and source on the web uses the same linear algebra equations. You cannot read about the Kalman filter without seeing\n",
"\n",

18639
01_g-h_filter.ipynb
View File

File diff suppressed because one or more lines are too long

40
02_Discrete_Bayes.ipynb
View File

File diff suppressed because one or more lines are too long

5950
04_Gaussians.ipynb
View File

File diff suppressed because one or more lines are too long

16408
05_Kalman_Filters.ipynb
View File

File diff suppressed because one or more lines are too long

25326
06_Multivariate_Kalman_Filters.ipynb
View File

File diff suppressed because one or more lines are too long

1293
07_Kalman_Filter_Math.ipynb
View File

File diff suppressed because one or more lines are too long

29532
08_Designing_Kalman_Filters.ipynb
View File

File diff suppressed because one or more lines are too long

4433
09_Nonlinear_Filtering.ipynb
View File

File diff suppressed because one or more lines are too long

15160
10_Unscented_Kalman_Filter.ipynb
View File

File diff suppressed because one or more lines are too long

5133
11_Extended_Kalman_Filters.ipynb
View File

File diff suppressed because one or more lines are too long

794
12_Designing_Nonlinear_Kalman_Filters.ipynb
View File

File diff suppressed because one or more lines are too long

2326
13_Smoothing.ipynb
View File

File diff suppressed because one or more lines are too long

11518
14_Adaptive_Filtering.ipynb
View File

File diff suppressed because one or more lines are too long

421
15_HInfinity_Filters.ipynb
View File

File diff suppressed because one or more lines are too long

1369
16_Ensemble_Kalman_Filters.ipynb
View File

File diff suppressed because one or more lines are too long

26
Appendix_B_Symbols_and_Notations.ipynb
View File

@ -59,21 +59,21 @@
" }\n",
" .text_cell_render h2 {\n",
" font-weight: 200;\n",
" font-size: 20pt;\n",
" font-size: 16pt;\n",
" font-style: italic;\n",
" line-height: 100%;\n",
" color:#c76c0c;\n",
" margin-bottom: 0.5em;\n",
" margin-top: 1.5em;\n",
" display: block;\n",
" white-space: nowrap;\n",
" display: inline;\n",
" white-space: wrap;\n",
" } \n",
" h3 {\n",
" font-family: 'Open sans',verdana,arial,sans-serif;\n",
" }\n",
" .text_cell_render h3 {\n",
" font-weight: 300;\n",
" font-size: 18pt;\n",
" font-weight: 200;\n",
" font-size: 14pt;\n",
" line-height: 100%;\n",
" color:#d77c0c;\n",
" margin-bottom: 0.5em;\n",
@ -85,8 +85,8 @@
" font-family: 'Open sans',verdana,arial,sans-serif;\n",
" }\n",
" .text_cell_render h4 {\n",
" font-weight: 300;\n",
" font-size: 16pt;\n",
" font-weight: 100;\n",
" font-size: 14pt;\n",
" color:#d77c0c;\n",
" margin-bottom: 0.5em;\n",
" margin-top: 0.5em;\n",
@ -97,7 +97,7 @@
" font-family: 'Open sans',verdana,arial,sans-serif;\n",
" }\n",
" .text_cell_render h5 {\n",
" font-weight: 300;\n",
" font-weight: 200;\n",
" font-style: normal;\n",
" color: #1d3b84;\n",
" font-size: 16pt;\n",
@ -108,9 +108,9 @@
" }\n",
" div.text_cell_render{\n",
" font-family: 'Arimo',verdana,arial,sans-serif;\n",
" line-height: 135%;\n",
" font-size: 125%;\n",
" width:750px;\n",
" line-height: 125%;\n",
" font-size: 120%;\n",
" width:740px;\n",
" margin-left:auto;\n",
" margin-right:auto;\n",
" text-align:justify;\n",
@ -228,6 +228,7 @@
" },\n",
" displayAlign: 'center', // Change this to 'center' to center equations.\n",
" \"HTML-CSS\": {\n",
" scale:85,\n",
" styles: {'.MathJax_Display': {\"margin\": 4}}\n",
" }\n",
" });\n",
@ -264,6 +265,7 @@
"source": [
"This is just notes at this point. \n",
"\n",
"\n",
"## State\n",
"\n",
"$x$ (Brookner, Zarchan, Brown)\n",
@ -412,7 +414,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.4.2"
"version": "3.4.3"
}
},
"nbformat": 4,

2
styles/custom2.css
View File

@ -209,8 +209,6 @@
},
displayAlign: 'center', // Change this to 'center' to center equations.
"HTML-CSS": {
availableFonts: ["TeX"],
preferredFont: "TeX",
scale:85,
styles: {'.MathJax_Display': {"margin": 4}}
}