daviddoji/Kalman-and-Bayesian-Filters-in-Python
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Copy edit of preface.

Browse Source 
Trying to reduce my verbosity. Still a way to go...
This commit is contained in:
Roger Labbe 2015-07-16 08:50:20 -07:00
parent 60c17168b0
commit cc24dd6291
1 changed files with 18 additions and 43 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

61
00-Preface.ipynb
View File

@ -291,11 +291,11 @@
"\n",
"Sensors are noisy. The world is full of data and events that we want to measure and track, but we cannot rely on sensors to give us perfect information. I have a mechanical scale at home; if I weigh the same object multiple times I often get slightly different readings. \n",
"\n",
"In simple cases the solution is obvious. If my scale gives slightly different readings I can just take a few readings and average them. Or I can replace it with a more accurate scale. But what do we do when the sensor is very noisy, or the environment makes data collection difficult? We may be trying to track the movement of a low flying aircraft. We may want to create an autopilot for a drone, track the milk production of cows, or ensure that our farm tractor seeded the entire field. I do computer vision, and I need to track moving objects in images, and the computer vision algorithms create very noisy and unreliable results. \n",
"In simple cases the solution is obvious. If my scale gives slightly different readings I can just take a few readings and average them. Or I can replace it with a more accurate scale. But what do we do when the sensor is very noisy, or the environment makes data collection difficult? We may be trying to track the movement of a low flying aircraft. We may want to create an autopilot for a drone, or ensure that our farm tractor seeded the entire field. I do computer vision, and I need to track moving objects in images, and the computer vision algorithms create very noisy and unreliable results. \n",
"\n",
"This book teaches you how to solve these sorts of filtering problems. We use several different algorithms, but they are all based on Bayesian probability. Later in the book Bayesian probability will be introduced with mathematical rigor, but in simple terms it is merely determining what is likely to be true based on past information. For example, suppose you drive home late and see a masked person breaking the window of your neighbors house. You'd conclude that they are a burglar. But you know what your neighbor looks like, and despite the mask this person looks like him. You connot be sure, but your certainty that this is a crime lessens. Now suppose that you remember that this is October 31 - Halloween in the US, when we dress up in costumes. This provides a reason for the mask, and you form the hypothesis that your neighbor is locked outside of his house. Finally, suppose your wife pipes up with \"didn't they sell that house last week? Why is he trying to get inside\". This new piece of information seems to favor the crime hypothesis, and you call the police.\n",
"This book teaches you how to solve these sorts of filtering problems. I use many different algorithms, but they are all based on **Bayesian probability**. In simple terms Bayesian probability determines what is likely to be true based on past information. For example, suppose you drive home late and see a masked person breaking the window of your neighbors house. You'd conclude that they are a burglar. But you know what your neighbor looks like, and despite the mask this person looks like him. You connot be sure, but your certainty that this is a crime lessens. Now suppose that you remember that this is October 31 - Halloween in the US, when we dress up in costumes. This provides a reason for the mask, and you form the hypothesis that your neighbor locked himself out. Finally, your wife interjects \"didn't they sell that house last week? Why is he trying to get inside\". This new piece of information favors the crime hypothesis, and you call the police.\n",
"\n",
"There is more to Bayesian probability, but you have the main idea. Knowledge is uncertain, and we alter our beliefs based on the strength of the evidence. How does this apply to filtering? Say we are trying to track an aircraft that up to now has been flying straight and level. A sensor reports that it suddenly changed direction. Did it really do so, or is the sensor reporting bad data? It depends. If this is a jet fighter we may be very inclined to believe the report of a sudden maneuver. If it is a large, cumbersome passenger jet we might be very skeptical. If the sensor is extremely accurate then we would tend to believe it."
"There is more to Bayesian probability, but you have the main idea. Knowledge is uncertain, and we alter our beliefs based on the strength of the evidence. How does this apply to filtering? Say we are trying to track an aircraft that up to now has been flying straight and level. A sensor reports that it suddenly changed direction. Did it really turn, or is the data noisy? It depends. If this is a jet fighter we may be very inclined to believe the report of a sudden maneuver. If it is a large, cumbersome passenger jet we might be very skeptical. If the sensor is extremely accurate then we would tend to believe it. Our beliefs depend on the past and on our knowledge of the system we are tracking."
]
},
{
@ -308,12 +308,14 @@
"\n",
"The Kalman filter was invented by Rudolf Emil Kálmán to solve this sort of problem in a mathematically optimal way. The theory and applications are beautiful, but quite difficult to learn if you are not already well trained in topics such as signal processing, control theory, probability and statistics, and guidance and control theory. Its first use was on the Apollo missions to the moon, and since then it has been used in an enormous variety of domains. There are Kalman filters in aircraft, on submarines, on cruise missiles. Wall street uses them to track the market. They are used in robots, in IoT (Internet of Things) sensors, and in laboratory instruments. Chemical plants use them to control and monitor reactions. They are used in medical equipment to do things like medical imaging or to remove noise from cardiac signals. If it involves a sensor, a Kalman filter or a close relative to the Kalman filter is usually involved.\n",
"\n",
"I'm a software engineer that spent almost two decades in aerospace, and so I have always been 'bumping elbows' with the Kalman filter, but never implemented one. They always had a fearsome reputation for difficulty, and I did not have the requisite education. Everyone I met that did implement them had multiple graduate courses on the topic and extensive industrial experience with them. As I moved into solving tracking problems with computer vision the need to implement them myself became urgent. There are classic textbooks in the field, such as Grewal and Andrew's *Kalman Filtering*. But sitting down and trying to read many of these books is a dismal and trying experience if you do not have the background. Typically the first few chapters fly through several years of undergraduate math, blithely referring you to textbooks on, for example, Itō calculus, and presenting an entire semester's worth of statistics in a few brief paragraphs. These books are good textbooks for an upper undergraduate course, and an invaluable reference to researchers and professionals, but the going is truly difficult for the more casual reader. Notation is introduced without explanation, different texts use different words and variables names for the same concept, and the books are almost devoid of examples or worked problems. I often found myself able to parse the words and comprehend the mathematics of a definition, but had no idea as to what real world phenomena these words and math were attempting to describe. \"But what does that *mean?*\" was my repeated thought. Here are typical examples which once puzzled me:\n",
"I'm a software engineer that spent almost two decades in aerospace, and so I have always been 'bumping elbows' with the Kalman filter, but never implemented one. They always had a fearsome reputation for difficulty, and I did not have the requisite education. Everyone I met that did implement them had multiple graduate courses on the topic and extensive industrial experience with them. As I moved into solving tracking problems with computer vision the need to implement them myself became urgent. \n",
"\n",
"There are classic textbooks in the field, such as Grewal and Andrew's *Kalman Filtering*. But sitting down and trying to read many of these books is a dismal and trying experience if you do not have the necessary background. Typically the first few chapters fly through several years of undergraduate math, blithely referring you to textbooks on Itō calculus, and presenting an entire semester's worth of statistics in a few brief paragraphs. These books are good textbooks for an upper undergraduate or graduate level course, and an invaluable reference to researchers and professionals, but the going is truly difficult for the more casual reader. Notation is introduced without explanation, different texts use different words and variables names for the same concept, and the books are almost devoid of examples or worked problems. I often found myself able to parse the words and comprehend the mathematics of a definition, but had no idea as to what real world phenomena these words and math were attempting to describe. \"But what does that *mean?*\" was my repeated thought. Here are typical examples which once puzzled me:\n",
"\n",
"$$\\hat{x}_{k} = \\Phi_{k}\\hat{x}_{k-1} + G_ku_{k-1} + K_k[z_k - H\\Phi_{k}\\hat{x}_{k-1} - HG_ku_{k-1} ] \\\\\n",
"\\mathbf{P}_{k\\mid k} = (I - \\mathbf{K}_k \\mathbf{H}_{k})\\textrm{cov}(\\mathbf{x}_k - \\hat{\\mathbf{x}}_{k\\mid k-1})(I - \\mathbf{K}_k \\mathbf{H}_{k})^{\\text{T}} + \\mathbf{K}_k\\textrm{cov}(\\mathbf{v}_k )\\mathbf{K}_k^{\\text{T}}$$\n",
"\n",
"However, as I began to finally understand the Kalman filter I realized the underlying concepts are quite straightforward. A few simple probability rules, some intuition about how we integrate disparate knowledge to explain events in our everyday life and the core concepts of the Kalman filter are accessible. Kalman filters have a reputation for difficulty, but shorn of much of the formal terminology the beauty of the subject and of their math became clear to me, and I fell in love with the topic. \n",
"However, as I began to finally understand the Kalman filter I realized the underlying concepts are quite straightforward. A few simple probability rules, some intuition about how we integrate disparate knowledge to explain events and the core concepts of the Kalman filter are accessible. Kalman filters have a reputation for difficulty, but shorn of much of the formal terminology the beauty of the subject and of their math became clear to me, and I fell in love with the topic. \n",
"\n",
"As I began to understand the math and theory more difficulties appeared. A book or paper will make some statement of fact and presents a graph as proof. Unfortunately, why the statement is true is not clear to me, and I didn't understand how to create the plot. Or maybe I wonder \"is this true if R=0?\" Or the author provides pseudocode at such a high level that the implementation is not obvious. Some books offer Matlab code, but I do not have a license to that expensive package. Finally, many books end each chapter with many useful exercises. Exercises which you need to understand if you want to implement Kalman filters for yourself, but exercises with no answers. If you are using the book in a classroom, perhaps this is okay, but it is terrible for the independent reader. I loathe that an author withholds information from me, presumably to avoid 'cheating' by the student in the classroom.\n",
"\n",
@ -321,13 +323,13 @@
"\n",
"I wrote this book to address all of those needs. This is not the book for you if you design radars for Raytheon. Go get a Masters or PhD at Georgia Tech, UW, or the like, because you'll need it. This book is for the hobbyist, the curious, and the working engineer that needs to filter or smooth data. \n",
"\n",
"This book is interactive. While you can read it online as static content, I urge you to use it as intended. It is written using Jupyter Notebook (formally known as IPython Notebook). This allows me to combine text, Python, and Python output in one place. Every plot, every piece of data in this book is generated from Python that is available to you inside the notebook. Want to double the value of a parameter? Click on the Python cell, change the parameter's value, and click 'Run'. A new plot or printed output will appear in the book. \n",
"This book is interactive. While you can read it online as static content, I urge you to use it as intended. It is written using Jupyter Notebook (formally known as IPython Notebook). This allows me to combine text, math, Python, and Python output in one place. Every plot, every piece of data in this book is generated from Python inside the notebook. Want to double the value of a parameter? Just change the parameter's value, and press CTRL-ENTER. A new plot or printed output will appear. \n",
"\n",
"This book has exercises, but it also has the answers. I trust you. If you just need an answer, go ahead and read the answer. If you want to internalize this knowledge, try to implement the exercise before you read the answer. \n",
"This book has exercises, but it also has the answers. I trust you. If you just need an answer, go ahead and read the answer. If you want to internalize this knowledge, try to implement the exercise before you read the answer. Since the book is interactive, you enter and run your solution inside the book - you don't have to move to a different environment, or deal with importing a bunch of stuff before starting.\n",
"\n",
"This book has supporting libraries for computing statistics, plotting various things related to filters, and for the various filters that we cover. This does require a strong caveat; most of the code is written for didactic purposes. It is rare that I chose the most efficient solution (which often obscures the intent of the code), and in the first parts of the book I did not concern myself with numerical stability. This is important to understand - Kalman filters in aircraft are carefully designed and implemented to be numerically stable; the naive implementation is not stable in many cases. If you are serious about Kalman filters this book will not be the last book you need. My intention is to introduce you to the concepts and mathematics, and to get you to the point where the textbooks are approachable.\n",
"If you are a hobbiest this book should provide everything you need. If you are serious about Kalman filters you'll need more. My intention is to introduce you to the concepts and mathematics, and to get you to the point where textbooks and papers are approachable.\n",
"\n",
"Finally, this book is free. I've spent several thousand dollars on Kalman filtering books. I cannot believe they are within the reach of someone in a depressed economy or a financially struggling student. I have gained so much from free software like Python, and free books like those from Allen B. Downey [here](http://www.greenteapress.com/) [1]. It's time to repay that. So, the book is free, it is hosted on free servers at GitHub, and it uses only free and open software such as IPython and MathJax. "
"This book is free. I've spent several thousand dollars on Kalman filtering books. I cannot believe they are within the reach of someone in a depressed economy or a financially struggling student. I have gained so much from free software like Python, and free books like those from Allen B. Downey [here](http://www.greenteapress.com/) [1]. It's time to repay that. So, the book is free, it is hosted on free servers at GitHub, and it uses only free and open software such as IPython and MathJax."
]
},
{
@ -391,34 +393,7 @@
"2. All of the notebooks in the book are listed, but in order of time, not name. At the upper right is an down arrow button with 1..9 beside it. This will resort by name. Click it.\n",
"\n",
" \n",
"Now all of the Notebooks are listed in order. To run a notebook, just click on it's name. You'll be in a fully functional Notebook, hosted on the sagemath servers. You can run code, play with the animations, change code, heck, change the text. Do whatever you want. It's yours, free. Make a change and push it back to my repo. \n",
"\n",
"\n",
"### Updating the Book if I Made a Change.\n",
"\n",
"1. Open a terminal as explained above. You can open the terminal you made during installation. They stay around until you delete them, even if you log out.\n",
"\n",
"2. from the home directory, enter the command\n",
"\n",
" git pull\n",
" \n",
"You may not enjoy life if you made a lot of changes to the book as git will try to merge both of our changes together. If you are good at `git` you know what to do. You can blow away all of your changes with the following commands (instead of running `git pull`)\n",
"\n",
" git fetch --all\n",
" git reset --hard origin/master\n",
"\n",
"If you don't care about saving the changes you made you may just opt to delete the directory and reload it. You'd do that with\n",
"\n",
" rm -rf Kalman-and-Bayesian-Filters-in-Python\n",
" git clone --depth=1 https://github.com/rlabbe/Kalman-and-Bayesian-Filters-in-Python.git\n",
" cd Kalman-and-Bayesian-Filters-in-Python/\n",
" git clone --depth=1 https://github.com/rlabbe/filterpy.git\n",
"\n",
"I'm not going to teach you git, so I'll stop here.\n",
"\n",
"I grant you this is not exactly painless, but this does allow you full interactivity, in the cloud, for free, with a free linux session!\n",
"\n",
"If you do go this route, spend some time in the FAQ. You can share your Notebooks with others (\"hey, look at this Julia version I did!\") and a lot more that I haven't discussed. Jupyter Notebooks is just one of several features they offer."
"Now all of the Notebooks are listed in order. To run a notebook, just click on it's name. You'll be in a fully functional Notebook, hosted on the sagemath servers. You can run code, play with the animations, change code, heck, change the text. Do whatever you want. It's yours, free. Make a change and push it back to my repo. "
]
},
{
@ -456,9 +431,9 @@
"\n",
"You can clone it to your hard drive with the command \n",
"\n",
"`git clone https://github.com/rlabbe/Kalman-and-Bayesian-Filters-in-Python.git`\n",
"`git clone --depth=1 https://github.com/rlabbe/Kalman-and-Bayesian-Filters-in-Python.git`\n",
" \n",
"This will create a directory named `Kalman-and-Bayesian-Filters-in-Python`. Navigate to the directory, and run IPython notebook with the command \n",
"This will create a directory named `Kalman-and-Bayesian-Filters-in-Python`. The `depth` parameter just gets you the latest version. Unless you need to see my entire commit history this is a lot faster and saves space. Navigate to the directory, and run IPython notebook with the command\n",
"\n",
" ipython notebook\n",
"\n",
@ -529,15 +504,15 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"I am first and foremost a programmer. Most Kalman filtering and other engineering texts are written by mathematicians or engineers. As a result, the software is usually not production quality. I will take Paul Zarchan's book *Fundamentals of Kalman Filtering* as an example. This is a fantastic book and it belongs in your library. But the code is Fortran without any subroutines beyond calls to functions like `MATMUL`. This means that Kalman filters are re-implemented over and over again throughout the book. The same listing mixes simulation code with filtering code, so until you become aware of the author's naming style it can be difficult to ascertain what code is the filter and what code is the simulation. Some chapters implement the same filter in subtly different ways, and uses bold text to highlight the few lines that changed. If he needs to use Runge Kutta, that is just embedded in the code, without comments. \n",
"Most Kalman filtering and other engineering texts are written by mathematicians or academics. When there is software(rarely) , it is not production quality. I will take Paul Zarchan's book *Fundamentals of Kalman Filtering* as an example. This is a fantastic book and it belongs in your library, and is one of the few books that provides full source for every example and chart. But the code is Fortran without any subroutines beyond calls to functions like `MATMUL`. Kalman filters are re-implemented throughout the book. The same listing mixes simulation with filtering code, making it hard to distinguish them. Some chapters implement the same filter in subtly different ways, and uses bold text to highlight the few lines that changed. If Runge Kutta is needed it is embedded in the code, without comments. \n",
"\n",
"There's a better way. If I want to perform an SVD I call `svd()`, I do not embed an SVD implementation in my code. I don't want to implement SVD multiple times and debug it several times. if I do find a bug, I can fix it once and be assured that it now works across all my different projects. And, it is readable. It is rare that I care about the implementation of SVD.\n",
"There's a better way. If I want to perform Runge Kutta I call `ode45`, I do not embed an Runge Kutta implementation in my code. I don't want to implement Runge Kutta multiple times and debug it several times. if I do find a bug, I can fix it once and be assured that it now works across all my different projects. And, it is readable. It is rare that I care about the implementation of Runge Kutta.\n",
"\n",
"This is a textbook on Kalman filtering, and you can argue that we *do* care about the implementation of Kalman filters. That is true, but you will find out the code that performs the filtering amounts to 10 or so lines of code. The code to implement the math is fairly trivial. Most of the work that Kalman filters requires is the design of the matrices that get fed into the math engine.\n",
"\n",
"A possible downside is that the equations that perform the filtering are hidden behind functions, which we could argue is a loss in a pedagogical text. I argue the converse. I want you to learn how to use Kalman filters in the real world, for real projects, and you shouldn't be cutting and pasting established algorithms all over the place. If you want to use Runge Kutta you call `ode45`, you don't re-implement it from scratch. We will do the same here.\n",
"A possible downside is that the equations that perform the filtering are hidden behind functions, which we could argue is a loss in a pedagogical text. I argue the converse. I want you to learn how to use Kalman filters in the real world, for real projects, and you shouldn't be cutting and pasting established algorithms all over the place.\n",
"\n",
"I use Python classes. I do not use inheritance or virtual functions or any of that sort of object oriented design. I use classes as a way to organize the data that the filters require. For example, the `KalmanFilter` class stores matrices called `x`, `P`, `R`, `Q`. I've seen procedural libraries for Kalman filters, and they require the programmer to maintain all of those matrices. This perhaps isn't so bad for a toy program, but program a bank of Kalman filters and you will not enjoy having to manage all of those matrices and other associated data."
"I use Python classes. I mostly use classes as a way to organize the data that the filters require, not to implement OO features such as inheritence. For example, the `KalmanFilter` class stores matrices called `x`, `P`, `R`, `Q`. I've seen procedural libraries for Kalman filters, and they require the programmer to maintain all of those matrices. This perhaps isn't so bad for a toy program, but program a bank of Kalman filters and you will not enjoy having to manage all of those matrices and other associated data. I have derived from these classes occasionaly in my own work, and find it handy, but I don't want to force OO on people as I know many do not like it."
]
},
{