From 931cc6935b0b2c23ddbbd1983458243bf86cf73b Mon Sep 17 00:00:00 2001 From: Jeremy Howard Date: Sat, 29 Feb 2020 11:55:03 -0800 Subject: [PATCH] spelling --- .gitignore | 1 + 01_intro.ipynb | 205 +++++++++++++++++------ 02_production.ipynb | 398 +++++++++++++++++++++++++++++--------------- 3 files changed, 418 insertions(+), 186 deletions(-) create mode 100644 .gitignore diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..87620ac --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +.ipynb_checkpoints/ diff --git a/01_intro.ipynb b/01_intro.ipynb index 1452241..c15fa93 100644 --- a/01_intro.ipynb +++ b/01_intro.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "metadata": {}, "outputs": [], "source": [ @@ -53,6 +53,13 @@ "# Your deep learning journey" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "TK Add an introduction here. Todo when preface is settled" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -91,19 +98,26 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Deep learning has power, flexibility, and simplicity. That's why we believe it should be applied across many disciplines. These include the social and physical sciences, the arts, medicine, finance, scientific research, and much more. To give a personal example, despite having no background in medicine, Jeremy started Enlitic, a company that uses deep learning algorithms to diagnose illness and disease. And Enlitic now does better than doctors in certain cases.\n", + "Deep learning has power, flexibility, and simplicity. That's why we believe it should be applied across many disciplines. These include the social and physical sciences, the arts, medicine, finance, scientific research, and much more. To give a personal example, despite having no background in medicine, Jeremy started Enlitic, a company that uses deep learning algorithms to diagnose illness and disease. And Enlitic now does better than doctors in certain cases. TK Melissa: Give an example\n", "\n", "Here's a list of some of the thousands of tasks that deep learning (or methods heavily using deep learning) is now the best in the world at:\n", "\n", - "- NLP: answering questions; speech recognition; summarizing documents; classifying documents; finding names, dates, etc in documents; searching for articles mentioning a concept\n", - "- Computer vision: satellite and drone imagery interpretation (e.g. for disaster resilience); face recognition; image captioning; reading traffic signs; locating pedestrians and vehicles in autonomous vehicles\n", - "- Medicine: Finding anomalies in radiology images, including CT, MRI, and x-ray; counting features in pathology slides; measuring features in ultrasounds; diagnosing diabetic retinopathy\n", - "- Biology: folding proteins; classifying proteins; many genomics tasks, such as tumor-normal sequencing and classifying clinically actionable genetic mutations; cell classification; analyzing protein/protein interactions\n", - "- Image generation: Colorizing images; increasing image resolution; removing noise from images; Converting images to art in the style of famous artists\n", - "- Recommendation systems: web search; product recommendations; home page layout\n", + "- NLP:: answering questions; speech recognition; summarizing documents; classifying documents; finding names, dates, etc in documents; searching for articles mentioning a concept\n", + "- Computer vision:: satellite and drone imagery interpretation (e.g. for disaster resilience); face recognition; image captioning; reading traffic signs; locating pedestrians and vehicles in autonomous vehicles\n", + "- Medicine:: Finding anomalies in radiology images, including CT, MRI, and x-ray; counting features in pathology slides; measuring features in ultrasounds; diagnosing diabetic retinopathy\n", + "- Biology:: folding proteins; classifying proteins; many genomics tasks, such as tumor-normal sequencing and classifying clinically actionable genetic mutations; cell classification; analyzing protein/protein interactions\n", + "- Image generation:: Colorizing images; increasing image resolution; removing noise from images; Converting images to art in the style of famous artists\n", + "- Recommendation systems:: web search; product recommendations; home page layout\n", "- Playing games (better than humans and better than any other computer algorithm): Chess, Go, Most Atari videogames, many real-time strategy games\n", - "- Robotics: handling objects that are challenging to locate (e.g. transparent, shiny, lack of texture) or hard to pick up\n", - "- Other applications: financial and logistical forecasting; text to speech; much much more..." + "- Robotics:: handling objects that are challenging to locate (e.g. transparent, shiny, lack of texture) or hard to pick up\n", + "- Other applications:: financial and logistical forecasting; text to speech; much much more..." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Deep learning is based on a type of models called neural networks. Before we explain to you all about it, let's start with a bit of history." ] }, { @@ -126,7 +140,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The realised that a simplified model of a real neuron could be represented using simple addition and thresholding. Pitts was an autodidact who, by age 12, had received an offer to study at Cambridge with the great Bertrand Russell. He did not take up this invitation, and indeed throughout his life did not accept any offers of advanced degrees or positions of authority. Most of his famous work was done whilst he was homeless. Despite his lack of an officially recognized position, and increasing social isolation, his work with McCulloch was influential, and was picked up by a psychologist named Frank Rosenblatt." + "The realised that a simplified model of a real neuron could be represented using simple addition and thresholdingas shown in <>. Pitts was self-taught, and, by age 12, had received an offer to study at Cambridge with the great Bertrand Russell. He did not take up this invitation, and indeed throughout his life did not accept any offers of advanced degrees or positions of authority. Most of his famous work was done whilst he was homeless. Despite his lack of an officially recognized position, and increasing social isolation, his work with McCulloch was influential, and was picked up by a psychologist named Frank Rosenblatt." ] }, { @@ -153,6 +167,8 @@ "\n", "> : _…people are smarter than today s computers because the brain employs a basic computational architecture that is more suited to deal with a central aspect of the natural information processing tasks that people are so good at. …we will introduce a computational framework for modeling cognitive processes that seems… closer than other frameworks to the style of computation as it might be done by the brain._ (Parallel distributed processing, chapter 1)\n", "\n", + "TK Melissa: Tell the reader what the takeaways from this are in your own words, before you dive into the list of requirements.\n", + "\n", "It defined \"Parallel Distributed Processing\" as requiring:\n", "\n", "1. A set of *processing units*\n", @@ -272,7 +288,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Harvard professor David Perkins, who wrote Making Learning Whole, has much to say about this. The basic idea is to teach the *whole game*. That means that's if you're teaching baseball, you first take people to a baseball game or get them to play it. You don't teach them how to line thread into a ball, the physics of a parabola, or the coefficient of friction of a ball on a bat.\n", + "Harvard professor David Perkins, who wrote Making Learning Whole, has much to say about teaching. The basic idea is to teach the *whole game*. That means that's if you're teaching baseball, you first take people to a baseball game or get them to play it. You don't teach them how to line thread into a ball, the physics of a parabola, or the coefficient of friction of a ball on a bat.\n", "\n", "Paul Lockhart, a Columbia math PhD, former Brown professor, and K-12 math teacher, imagines in the influential essay A Mathematician's Lament a nightmare world where music and art are taught the way math is taught. Children would not be allowed to listen to or play music until they have spent over a decade mastering music notation and theory, spending classes transposing sheet music into a different key. In art class, students study colours and applicators, but aren't allowed to actually paint until college. Sound absurd? This is how math is taught–we require students to spend years doing rote memorization, and learning dry, disconnected *fundamentals* that we claim will pay off later, long after most of them quit the subject.\n", "\n", @@ -305,7 +321,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Your projects and your mindset" + "What you will need to succeed however is to apply what you learn in this book to a personal project and always perservere." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Your projects and your mindset" ] }, { @@ -314,9 +337,7 @@ "source": [ "Whether you're excited to identify if plants are diseased from pictures of their leaves, auto-generate knitting patterns, diagnose TB from x-rays, or determine when a raccoon is using your cat door, we will get you using deep learning on your own problems (via pre-trained models from others) as quickly as possible, and then will progressively drill into more details. You'll learn how to use deep learning to solve your own problems at state-of-the-art accuracy within the first 30 minutes of the next chapter! (And feel free to skip straight to there now if you're dying to get coding right away.) There is a pernicious myth out there that you need to have computing resources and datasets the size of those at Google to be able to do deep learning, and it's not true.\n", "\n", - "So, what sort of tasks make for good test cases? You could train your model to distinguish between Picasso and Monet paintings or to pick out pictures of your daughter instead of pictures of your son. It helps to focus on your hobbies and passions–setting yourself four of five little projects rather than striving to solve a big, grand problem tends to work better when you're getting started. Since it is easy to get stuck, trying to be too ambitious too early can often backfire. Then, once you've got the basics mastered, aim to complete something you're really proud of!\n", - "\n", - "Common character traits in the people that do well at deep learning include playfulness and curiosity. The late physicist Richard Feynman is an example of someone who we'd expect to be great at deep learning: his development of an understanding of the movement of subatomic particles came from his amusement at how plates wobble when they spin in the air." + "So, what sort of tasks make for good test cases? You could train your model to distinguish between Picasso and Monet paintings or to pick out pictures of your daughter instead of pictures of your son. It helps to focus on your hobbies and passions–setting yourself four of five little projects rather than striving to solve a big, grand problem tends to work better when you're getting started. Since it is easy to get stuck, trying to be too ambitious too early can often backfire. Then, once you've got the basics mastered, aim to complete something you're really proud of!" ] }, { @@ -326,6 +347,20 @@ "> J: Deep learning can be set to work on almost any problem. For instance, my first startup was a company called FastMail, which provided enhanced email services when it launched in 1999 (and still does to this day). In 2002 I set it up to use a primitive form of deep learning – single-layer neural networks – to help to categorise emails and stop customers from receiving spam." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Common character traits in the people that do well at deep learning include playfulness and curiosity. The late physicist Richard Feynman is an example of someone who we'd expect to be great at deep learning: his development of an understanding of the movement of subatomic particles came from his amusement at how plates wobble when they spin in the air." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's now focus on what you will learn, starting with the software." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -360,6 +395,13 @@ "Since the most important thing for learning deep learning is writing code and experimenting, it's important that you have a great platform for experimenting with code. The most popular programming experimentation platform is called Jupyter. This is what we will be using throughout this book. We will show you how you can use Jupyter to train and experiment with models and introspect every stage of the data pre-processing and model development pipeline. Jupyter is the most popular tool for doing data science in Python, for good reason. It is powerful, flexible, and easy to use. We think you will love it!" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's see it in practice and train our first model." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -367,6 +409,20 @@ "## Your first model" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As we said before, we will teach how to do things before we explain why they work. Following this top-down approach, we will begin by actually training an image classifier to recognize dogs and cats with almost 100% accuracy. To train this model and run our experiments, you will need some initial setup. Don't worry, it's not as hard as it looks like." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> s: Do not skip the setup part even if it looks intimidating at first, especially if you have little or no experience using things like a terminal or the command line. Most of that is actually not necessary and you will find that the easiest servers can be setup with just your usual web browser. It is crucial that you run your own experiments in parallel with this book in order to learn." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -486,7 +542,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": {}, "outputs": [ { @@ -505,10 +561,10 @@ " \n", " \n", " 0\n", - " 0.162706\n", - " 0.015182\n", - " 0.005413\n", - " 00:15\n", + " 0.167097\n", + " 0.032373\n", + " 0.008796\n", + " 00:14\n", " \n", " \n", "" @@ -536,9 +592,9 @@ " \n", " \n", " 0\n", - " 0.056831\n", - " 0.033615\n", - " 0.006766\n", + " 0.044406\n", + " 0.008025\n", + " 0.002706\n", " 00:18\n", " \n", " \n", @@ -591,7 +647,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, "metadata": {}, "outputs": [ { @@ -600,7 +656,7 @@ "2" ] }, - "execution_count": null, + "execution_count": 3, "metadata": {}, "output_type": "execute_result" } @@ -618,17 +674,17 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJcAAADACAIAAACGdmZhAADHOElEQVR4nEz9SZMlS5Ymhp1JVc3s3uvX3WN888vMl5mVWVVd1SXdaAjBrhZ0NwiSIhAA/AEk/wmF/BvcQMgFNxQhKMItNwTYKPYgXSxUVlbO+caIF4OHu9/BTFXPwIX5y+Zd+Cpc/Iapqeo533QQ4J99+vHV//5/97/9L//Ln/zf/y//h+fXx89+cNXbWWizzPlXv379q19/8+pm4zAgOIYDOHGULMv5dD7fEyKAM1MZByVU1XAnQvRw7ahWEKz2en+s84xmZOhqDJhSgggza605QhnHNJYyjrvLPSaMCA1XV2CilICoqnUHImHJQRgKQpJkQCB3QGREImQAcHczNTMl7+LuTohZUs6DsJiqdWNiEVE1bYvwEdf/RkSYR2gS0dogAiDcPaVksc25iKRuGoFSSm2mZkjMKROzh8fSyB0ptM/uLUuIULh572YGERiETEpuXYeUh5zb0uZlpiQo7EINrLlbeLScRJgoTIkiM7srhoE7RjAjExJdmIOaurk8ecb/1f/qn/3lX/75/+O//W9qna8ePa1VwYfjGV68uP3d79++fdvMN0gOEBER4W6G4O7OxEQIgMwYABjBgMgCGO5GiAFxPi+xNFfzbr1WAQpzQ4wIBFifkYabW/SOwq015hQIjgDIAAiAASiSGcmBANCdiBARzSxCCTmlBEDhEQCADhgRToQ5iamFmbtpqygOAeAaEGGI7uFa+yIiKQkBBqIH1dbcLKXETAwwpOQwEEkEhAcAIkApSTwBIiA7gHULMxGWRMRu3QHNzM16mLkZAgAgYgBTwpQkMXMqOTAsApiQKAEEIFh4BEYAGBEyIhIOeQQz9x5mEWERS2sBhAjILP/Vf/0XP/hh+X//q//2629+9cc/fF8tXr2Z27nd3LSvvrp/9cZYng8o7g4QABgeqrqcF7fGTMJEzAAPDwDUHIAIKQKDzNGbe1UwT0TAKcwBKQDDIQAAiCVb7+5gatStd60JSQiBnQARAtgdkQWRwSEChRgRCRAAIgIRAIIoHBwC3C1CER2JiAkdLJDcAkC7g7m7BRJad/ewlokgwJoignsEBCABRSA4YkRUNwp3Vwdw9wDArmUcRcgDArCvOx8BmJAQA4OQiMysWYeIAEBAIVT33jyxcASYAwBxAvcABCAg8jAPN2uBTiQESAgR0VqDCIj15UcIqGYBjkgAIc/fP5/nvz+8u33/2cXr12+++v3tcjof7ut8xnnJPTZDuRBaADoAMKFFuKm2ykwlpyQswoCBQGZeHcxNiIUZ1WpTynk+L/U8k4cgGTEJYQAFICAhmrtBrN/MIzxCCUUkSWbkQGJid3QA03CHdXciETETUUTAw/4DpIAIDMcIWvewG4cTBiIQOER4uCAQAoUDRGYqlLpqdwMAFhFJlKT2FhFL6613Ihw5l1yklFLKus7gbuDrOYFhDE7CSNG01Ta7tZwRMNY/HAiIDMgBruaE6EROhAAE6AEAAQEUxOGkgaaI68pzBFrvzIzAQAIPn0iSTL2rurm0+fMvX99f7za/+XU93922c01ctAvJNg370Dx3SDBjVCICYXftrQXEMOSSM2IQMlBYV23mahEB6OAA7gwUBtq0nhtoFxIaMiBihHsQYBBp7wZBKAChprW1wJEkYR6IEgYzyTzX3oyQcy7MrKpJOCVeH6a7ATgRIaHpw6kKgAGB5gH48I5AIIQIEBEDAgQAhKNXxwABYmHOSVJ2BOi9m3VzdWCm1jsEIjPnTIRuCOHgYeGIRIGZBQgCQ91ab9pnCxJhZPFwdABEJ/QgD1A3C0+cwtwiKImpERAhR3gLyIwIzgD8cOo7ITsCADiAG7gHZDCA7q7uMh/eFEk3r1+jI7hInhAyWDjm5m7YeCjYLHoHEkZGxPWJEGF41F4ZIACsKwQSYmJGRO/q2r315TyjxZizBahaXWoSwQBXI8Qksm4wYXDziA7nJdronJBCCBHIDN2T8HBxsX/65Ok0jct8vr2/6b2rNTcNCE4kCQEiwgP8YeHWI9cfqhQIDwgEJCAgRHcARAjESJKQGRB8XT6A1rWHIXEeyzCOcY4IMDMzE0ksSITR3dXh4fgEBQ8AIkRCD69dAZKIhIVDIKA7oEgRdnfOpYyTqbVlIeIIJCQhDooMLUQAiYWIWMEDXNWRiSQxMwsFhEEgkyTxCGFPpEJevAeBcBprMw9D74AumTkbepznhr2he4ABOEBYVwTstR3mMzEPkt0BCZnFWtfWQrUez3WudV6wa0IZh2xgEWCq2pUAzAMQutqpNhTiLGrOdbvZ5QCq1cJwPs9XF9ePHz95/OjpxX47lGF4mu+Ody9efPn6zcvWLKADMiEgBgAwE9FaSlC4W6iaYQQiIn63tqZdu5lhxCi5lEySam+1tW6mHt3NAZCQUcyAAhAJEYmICJk5IoQRgN3MvIe7E2nvxDgNxWwx68xs7rU3BM5ZAMUDiEUQAnhuJsypTNo6cIIAc2CUadoB1NobIAkLYnQORAZKSEzIHuGBxOuNGQQg2AYNQCVXi8AeEW6EQawknQjCnBwTRiAGWpgxIVNi5lCzrmbGRIgEAW5h9l2l6FBbV1U3B3OCiGAUREQSEUBCZGZipmSgquDuCEiJ8nJc8m4Dzlbj049+eLG9nsZpKKN2OlnXjkPZffThZ4Ty1Te/d0CEUG1I62HK6xNHJEQyCAB72JuITGhdrVeHYEbhBMjdNZr33lutTR1YkiQgASIkCntYPgByd1U19/VMZggAN1fv6kyEiGbqnRHXa1uI3ELdzSAlTiKMaO6q3rVhAAKEOwUgQAJOzIlFwQUJHAIIMEQSIgcx0bqeiOAOjhHmHhFijQEw3DEwwk07kTOjsCMZWDfQiCyZ156qu4qwsBCghyNSlpxTJhYEcHAEQgYCAg9iRiQkIhECREQPX5/wQ69ApKYkIgDuyoxSMimOw4AdCub3P/zgow++Hy4BYBranQgJIqW02+6ePtO7w927229b9JSBECMcwhFxXTAiZoYACDcAIwIg7LaoKRIKJSLqptrMIRAliJAAkAiFSYAYEAEAmZiZiBDW28kdgBEQPMApTEOjQcpZw/p8TomQE0QI8zgMrZl5MHGiBGqA5AGOAYSACIAP7RBGAAgwsLCkiPAIBBhSXo9/RIqHC12ALNwN0DyEgtf6FRkjIqIzQyIUDAwIDzSPAGQiJDNDiCSMEK4dkIZSImdEjEAghAiLIEAHdwCS5AHmQYjCKRDDOgQ5xHpRuUVXlQyKoRZBIRpRlQYQoSePn330wffckhk6kFkPgDwwUu6dWl2Gsnv/+Sc3N69a78O4AVB3cA+EtfFDZHKPCPBY60UQWuEBRgRVUzNiqt4BUJhFZO1OIci6I6FwEibntSgWEWJCBDTTCMMIgCAIwdAIimB38GBAIqq9YeAwjkKpdiNidIgeQICIxATMgBhgDugBCOjIhuwBRAwYECGMIklVAwDcEeK7DisiHMMxTJjAXD2U0IksXIWQENGRAyGyuDivd6wB+npOQYSGoTuzEJKrAnEaBgDwUELU1mc7L7XOddHe1343IgyABMPd3cODEInYAczD3V015nmafL47fvDkk4/f+9jmaL2lYYeBHgDk4VibgeRWtZTxyeNnm83+eG7MomoAGBGAyEjEiVISSRAO6BARodqqAyIJEZipqQ1DZkB3twDzQERmdlvBg/U6SspBRMRETESEAKZm2sE9XMFDiEgAIgRxzCnCyQPN1TuUTWIJd/fACAIMcwNXc+sQiEgYHkiILCQixNWwmQFAYgKUWC91dw/DAABHwhVMIHdwF6Lo2s0qQLA4kTMRBcGKtYEwcmftqmtvQAQRIcRG1paqXRnJzYIYUxHhUNTwqv18Pp/r0t2ZGRBab62bZyEEAlz/CCJGhLkFQMoZGANRl+WDZx89vX4sQMgiMngwAIogcNAK/gDlPECgemw3u9buW+sRth51TJzTEMRmQOvBigEY4VDr3M0jLJOQsGPUXpkEAd2cAEVyGAgKZWIURMYAZiFCRPIIN+OI3lW1uXZwZwJmYsquhkKbYTjNJzMVFouw3p0hPMyBA0YkAFjXIzA8AmLtkggI1C0CKA1tmcOVidRMmyYhcEOzCAcICsoEgGyOACQOM1AjiCScRMAjIRMArNd3RLcWUFLO2ru7c0I1b83MIKQQYjOXkinzGSosJohoUe8P7XCCueUgJAIAEBIMB0Rfa0XC8PU0aV3HzVRVBVM3093lJz/+RxfXz7XJOF6FJ0A5zWf3RRIDWXjvoUg8q0pKZXOJxzsga+3s3nIpwFyjYyCEhHpiYAKts7aZtVHvSJiQA9gcWoitVVZhQgpApwjAZsYIKVGAsy4szLwij009IDojEIEHEUjmoREqKxEFhOS1JbaEiACtVQ8PgoYJeOPhEYwQae11VkgowglZhFgyBgh15e4REN08SxIUlMQRGObuDoWZURCJBNn5YacDBiGAOzAJEVoYuFESDYRggIgwkRTREZyQ3B0Q1+YjJRbEpr1Vtbme7u/1vHBg+EPpKSLEYOaMyIDAjg7hgeBZJDyEREh6t48+/UmZrpumknc573oDtSBMTG5Ww1sqHBhq6mCUhDgHCBEhCYQjSmCYGaEzgXXFCETTVrUu4UYRuMIAAExpGHduBgCAHAhmhkSAEIGYmIu4u5/PjkZI7l21RhghiyTk5EaIOaVByQDXiw6Zc7i6ahC4m1tzAkB0iHM44YrForu7uanhA3JHBgDuBdezlBZT8wiWZm5AjOCO6BEWYYYMTMwcIrAiHRFuQETMvB7/QZAEIoSl9WquAMFMIowIRNR7X5ZlRXwDPSqAIARo7/W8tKWamUeEmpuvUOcK2iARrVjZetAjlpKXplIG9bjYX/3w+3+UeQzjcbsTGdxcredSEvFcdelNUoqwCFBQ7bB2K2bNzQICMeChekCiAHZwUzdzU1MzQ+IIbN2JhTmZW5hFOHEgkxCmJERkZMSE4a49nMwIANxBjSLWJSFmQsoI0h5wWyRmBgw1N2QkZDJXNANwJDYDd6UkhASwFs9uYYLCIrQiD+EWwCJAoeGIkVOyrgErRL4ijtg5gJwJCEPAOoXH2g0iMMswjK326i4sxFS7qTkRAiIzi8jDRfvdBwCIyE3NVACFyQRFKDBMVYiEJcJXqCzCwDmIAdZ6xnMZPAKZu7o6/PRPfjiWq3G4LnmT00WtRjIWKk0XEiaZ4rz0thBzyskMAHyaxmEox+MZIogIIMIj3MI00ImBGb2bau2mEJBQHMgcgIQphelKFAQ6MSNi5gQIQBEeEa6tUyAGYxAgkjAgqvbuZCxMQpQASCEikNZDDSOQgCjnhOAZsLtaeLgjERLxun3MAEGSrM8xVjwZwM0kJWYUJ4RImRtYeKADWAASU3TQCKAgdBRQFSQSiqC1jOyqS2u1q0gm5taUwIl4LW/d3cx676q6tn3uzsws3NXAggEECdzDo+RcJBGRqS11MdeIMFMIRQQMWEs+C5RUTovurx599tlP0cdNeTSOOzUObykP3brWM7iWMW03m/NsDsgsajMAbsZpKKVVYRkQjQh7b9q7u1IAI3qYtdZ6NY+cC1Ii4HAASkh5zKyd/IHuCndrbSFCMxMREYaSsSOzIGM4AzgICY/r/kUpwILAbj3CNdARCRiJACOIAyg4BQKEIwTHuuceqKqVETKztURf+2jDlb5jEWYIQkDw9emHO0ZEICUIQiIEAFn5UkBs6mqm3mqzQCJOS9folvMAama2/rGIqLWqKjOnlFTV3SMgSfK+nE8nbx27M0BiGnIiJPSwcIBgRMnk7hCACMJILA6eh103IpYffPbTq0fvQzwe0kWdAyVNm8kiBCX8UHuzUCRLIt2B0MEDIlJiEV4pF5HEjIQgLMBuPptpaz3UiClJEikeDMDEgigeSO5rA4FI8d2RFQFro0nEKSFzWSsAiK7uoSA5hUd310BBBiDOhQLCHdxRKDFBhIXW1k5zdfSUiwigr5vQ/rArAGB9mBGx8snuqN0EURIjhodlQgtwRyfCAAwqyYOAiQNCci6q6rAeb0CyolgYsIK7SMwEHKYrNFlrXdcSEVcQg5kRoLfqrYEZqGmrocqErqrrW+auvasFE+WcmBAgCJBYLIRY7m9Pl9cf/PDHf4Y8jXIJJr01sJi2AyLUPk+7i353mufTdkNugQDaOhNShBBvxuGOwM3CEVlKFlU3aIjkQQZubok45YzIYQQoSQbkDADgqr2vx0lAIFPOufceDtqNUIkokEsuDm4VEcwBzMUiLBwd3SJCizAhIZGwMKGvdK5D6wpAzPxwXROuK9dai4ic8wpjrT+Z2c3YwsNX+sysIYBgsKCzhEIEIDBHd10BKRINWrpHABIhU0r5oQc3A7AI6FoLrkwnrDfi+u6srR4illKQKOqM5hRgZvPx3E9zYkHi3ntAPMDQEREhLDkzrd+b0uHY7u7PavLp93/69Nn3Ihg6LdY8iAnNopp2M1txZE8YhkHM0K2nzCyECKVkZnLH9UoEwNZatxnZXNVUI8wdzYxZfIXYmFEEINiTOxKTB7hDztk8PJAl01odBQij5OwAHJgdHMg8KJzRgQCBDD0APMLdaT3E1QDW19gkJc5s4RCQVmYR0d1brevDhAg3i3gAZbo7gJlZuHedmZmZkiTC7IkiECmRLr1XVcMIqT3UV1UAIRKyUESgc7h6oNtKca6ftYb9w4oiYkpJRADQrWkFUI3eyZ0JBSHMwi0iSDjnVApmSUho5l1rAAJaVzyd+kef/ugf/Pk/KeVKlUcZzudmanmY8jCEKgUdDmeiVPI4n98KOhaEiJUKR4gwQw9CgIAwD3S3Fu6A3loLB6bExB4B4UCEjEHo3t3BkTBlQGqtRUCipKYohYnWW7+25tCint2hdgNKyEjM7A7huELFoYmZAFYOPRDx4XjTiEAIBEgi6EEWEZCYMWU096YkiYktoPWGiDklgMBgACSEzCKMiMhILByYAAQlSycOXKyamRhmEAZEjwBfZSUAgMIEQRDAEonEANZadH1b6LsPIj7IkHqvx/N8OFnr4c4PFMeKGAEACDOLYFC4OXgEeoCHAwyb7fRHP/2HH330w/OZkIZE5N5bU1gyjVOeplCYNpv5vNTTEZxUm7kFey4YTmZhZkiIAb138+D17RUKUCUD8CSyXgFEFEgkTIK6vvwu3ZQJkDNEqCNxcXcHXL9koDiCgptDAAgLkQQQowIEBhIAB6IbEGZhNAw3cKdwRsqMvgrPIsDBuqlZdCUiIVZXsJVqBgpg5iFlHhO6QXSEIBYEd1NwMiVfS1mHxIKSUA0BhWRCMETUVs27uhMGYQRGYgQIYVgvtnX9Vqa0lCIi644EgN57b91ab7VGV+8K3QDpD3sXkQAAArV7gBECEQcEGNamz95//tlnP0EYHWIoOz3NiD6OJYTm+XTuzdGTkHu01hOJAXWta0Fu5hy4rtpaGYTH2hGZeTcDoPU/jg8UhzgyMAA7AiFiGLWugphzBkQHMHezAIiUUioljxSwCAsnghCREQJ7VzfkgAdABENNIVBEAtF7uCsSikAAd9eVBliRBUE0VU65rFexmgeYGXhQBHiYObr1eoaoWQBDV2YlIHqouQR22U4GZMghJMEZVn7V3bsTJYAeEQgeoe6qjhHFzNe9J8LMnJIQ4Sp5MbOlLn2ZTRe3TghJ0AMxgjBYeC2dIsKdIiSCgTwCVLU2s+Dr66dPHj+/vT2W8TFE5lS5Qy553F+24GNrhKi9JsnTuJkPc8KUqGmswrJg5LHkJGIdMydkIMTe21Jbd4cgDDeInDmXrGsr6YpOQABEYZzyICIr4pFLXpYFzQBAJEkSQFQ7AzkBYTAhmjoFZZZwIIpVR6TNzKNHgJmrMgIDegSBE4RDIHpC2U0bSbwsdV0YJUZEVYNwJmEgV1NzBG+9cvRRBDEggCAMEQIMAAEWcw3vCBEg4gsARGgKzYJoHcGZIBzc1j6YGwImYRFCEQ4iDg81DzdEcLda6/k026nWagIkQOAAAZgEiN29hboBh43hgAEOYymFUwJYNO82O1U7d49B2XVbhn44m2thCCLhnIrc3S65FE6b4/m1WUNmhLSSqEgsZaAktgRmJooIU9Vmc4QzETMnScTFgNVdHZobg1JGICJOOWUhjggiZkpDRl+xpYimEeFJMgAhMjEGmrqyEJi7WiDFKv8IDLemNWKF2ygikBBIwCHChErOg6SLnFNw7b25KSURoagVEAGCEIGAETw8l1GooBDhyuSRAxNTZg4ANzVVU3V3yX4EiN47swzj2FtzdyYhyYLi4UTUyRzYgDQIAqFTrFcEhHCYWe9LrS06IWQ3d0AwcA9zZ6cOoEDBDBQTmyQ5HE56tghJNHHZbDfb81I57xrqOCJA5jLQMBrTqS6HZR5p9ALO2HpPF+X+3R0rr+WFA/vaHos0N8nhK4mWkZpJmAcxC0tWI20BkgPRXQmFkiBwliHc0YOJGDm6oYcQB2FXMzdABBMzgIQo4NiVWw/Q6O6WMAtJ7+621jLAzEhUzRChSCEKCgLzTAPLdEaZO5hjU7euiJBW6VdKqxQGEcI6AuRhCoDFfChDybm1FqoQHma9N8QeEWiKERLAiMjCKQlzxiK1VkQWSVyotT6f57wb1i3lvuooNVzNlNEDQdt5mY9tOWPtRJhWIHEVPHYLCBQSJhTOCOSOQNvNrnscj5o3+fLJ88vHz1OZ6gIbGTdpQuucuLaWVIdSTnW+fXeznVKA9lpzknEY6/kcgQTMCCuwm5gEgfxB0M0RCSQBBzKBoKK5QTAzEQuzsCQGDgBicAgIg1WYgxTozIRMwAFGAdDPhiu9vKr7DM3M1MEhyMMdMVS7hzHLQw3oHgCq/UEkgAgIpq32Gu7hAeFuHcIyD8LMzBjWe++9B8Ja+WvvRDSUknMhImqt965q7uDWViRPEAUlAQALOGAzBwiU5BC6KhKRFAKalTExi/XWWhckprBWHc0izqfb4/2NLvPgJMhZRJAMgoCJ0sqFO3pYR0QGOB9PMm553JhWL9PHP/rp048+3Wwu52XOLoNzSgAQ797dtIj906ePrvZfvTgc7o7bUSLCet9MG2hNdSFI4iARzDCmVFjMFg+13jw8mnkwizBlosQojkQkzMXAEAgUuqmlMwIQgBs0rWtRHZAIxMM9LAByHiDAPXQxIATkkkuIhhkFgBmEISEBr500AKzA1opTrp9w1+jWq5knkZwzphzuOQkhEaI7ERCT5JLcXdUQRST17uY1ImIVTTDkzBgGq8w4XFByQACiqc21EdIwDhFhq0CXJA2bYz0DcZLUW+/LzDnlzK7u2jA8dA5dOEIgEoEwChFDODEiqftDEYGAqr0pIN0fz9Fxunr2g5/++Sc//ZPLZ+9v8/XxcIsGomjYWTgl7lrPp9vd5eVuym/evLVUhPz2eNhOgwiEBoeBOTomoSlJxqiq4WrLEuEUSFRKniQPxAUpBxIAoYh6BwgLQwOl8yo2BwizbqoI6CBkycAdARDHchWBVRc3DUBJDA4YTATobuiInlMChD+QBCLCzA8gM/PabiGEDGyGTCjkARiO4WauBkhIOZeRxATnefawnAsxz0sVSUQUAbo2Eh7bcQT31nvvLiAc7sTs7t2dGVd2HhEtwiEwyQAjBHi3RCiDRBi4u87WFxYEq5kcGVgdwsAVaOWCH+r2Va75APIZpnEQoMsPP/nkJ3/2/Z/8BcnFYVaG3pZ6Os20WMuLgadMSz+/enlXBrJ2Cl3AgMGykLaFXDM7uro6WGSSgohdvVYAx9YZEWRM0zYPI3LxQF0PNWYEUF8FuhABDOrWTQ3MIQIDEMHU3JojUBIRESLkxEla7xqGBLpCkuHh5qpmHQFJZBU9r1K5dSFXnPI7dWUkBid0s947AjIJAhs4xMobESCahnByCwgilHDV7iIEgKag6u7ee18FaUmSqK3aOCZOSB2Jl9pzppTY3FQ9IoSTtta0CiET1mWxufX5ZLoEg9UFV5WKr76j8O9EPg/uHIeAdWsHiTSLx+9/8I//6T9/8smPlijfvj69XObHm3p6ffCj+nmGq+hhwOLh727efPDBo9BjYaNoRHaxHc6ng0cncAKlcAxCddOqywzaEQK7DkOJnCylhhDhzVzDgEgwwlytQQS6BSrbssyL1oYBiZgQe2tAGERByEnKUNp4LsM2iQQAOgMBE60vuYWufbQDIvMKLKzA24p2/QHqAkCM6tiYRXuHoFxG4RSBgqIduqkuCzJByivAv3rHgMjCV17bYJVs0Ly0xCwpSWapbSmlIDIipCQejgSAHvBA7XoYOgkxMTMGeAjEvJxdG4XX46xzIwNQYEQWQaQIXxdUtbtHJnQPBsy5IOaUx/c//cHH3/8xTNf3N+d5ttcvX8/DspNxMyXAmoZBexfx7W771de/+fLzX3zvex9fXZBZC/Nl7h6V2ZhAJIgCvNfF6vmodcEwgkiEm6Foye9cdamUBkNyZhZavKk2puh16UtNgtBvz+ezN00sHdG61taISXJ2DEcYp4l2+10ESWrdOGeRYhau3ayDexYhsNYfmJ8HaPQ7kOtBW2SmqmEtqIdgUxtKHsbBLHr3lBK4abcISFIYVw9heHhrzc0l/cGh8R/wUBRGpkAUInS3rs3dAONBJe8rQxJIkCWBAgQIMYN3t7acrXcw867Rg10GCiggSgHoHg7RTQNBSFYYOiIgvIHL5fb9D7/3/ic/GDd75XEq3JZvtbbNo/H9J082KQ0iB1oup0fH47HrcRj889//7Pmz8epyB8BtCa2a2MZSEiNxLPP9svg4sGnLiU11GgacRvJwiLkePYLADLi5A5KZg2tC9Na0L76EzbemFmrLKgj1UFUgNBENNwhfFsZ9ylSmDZAgmJuuGJZ1AzDGYCJiMA8zyznnnL/j7B4W0t3dLQIDxQJzmrram3d3zImQm9lKVCCihtflLMwAoKsIIZH5arskSZyyuLsZASIQI7NAmKu79ohV6LhuvzBXj1VmJ7BChYQEGKZtWax38rAW3oA9JS6IEOxqtmr0mrkHKEOooeBKBTrBzJz2l3l7gTRwJDZlo+uLfckkIzx6tmeMeuplwAAGpGdPdvP51fHu26n0cShCXUQ3G9kMg/fe+mzRGcDcAP3yYjMvuBlGAry7vQVu3JurhZsG1tYUAgE4ItygV2tVu0LXlR321kGdEMEMCMFjPb2ARPu9w1ZoYCRHd6uurqpuxgiAQEhZqKpH+Iotr/tv3TXrvlyNBl0JEMchqUVTnTYcxF0ViSRLBHTtCMEiARBqAcEsMoh2dXckAnC1bhE9gNU5RLLQgyIfCREhoqv/QU6BiAY2ypYiVjqREZiod+212WLebe3nRRjQLEIBgsjRNdwMGNkxiQADpmlz/eknl++9N+4uS5ncini73lzkq1T76c27bzaX4N7c6e3tbWLOApLs6mo0PdSZBEfzHr4AKMnQtDVbONEwJvBFGDe7DQuim7auvZmZLLX3jjokZjMl83DjCF/maI166601R1pZJbWVJzE3JEa1FfhCFrOj+9Fh9FjbBwKg73x4axsRxCxCqj0i1rZvLWoevHmw1gnokRDRIK0GsjyOHoYIDqAU7t5DB6KV5grQVdpNxMQR4AEaEYgWSN1czROhMAIBrK8PAKhqN32QxyNCBCMwIRj0ruDN1RjJu7W5R/foEB6IqO4avWo3BBR24nUrQ2AQBUA4cMqPP/7o8YcfbPZXIqN3SsEjJdWWE1KOU7+n5H2G8/GYs1Q0t6VIJDGI2hXc1KyaqkVDwlwSIeWBtWEZSzgNRc6HY6/tYn9xOp3xdPJliVwxJwxgt4hAVTsdWDWBY9cOpfcOFplFiMOcQBCQgSEcLcjQvZovFvPKA5oxIBMirtgrqGnYKkler7S1xEP8w891RR1I8qaUQqsqlyHAFcAJfRXDuDsBMQU6IpQhr9xytwYAQKGuCMBChhmgqzkCiHl/kB5bAKCZQjgiM7EABgATmfZe+1KP0Ku1udcGAMwUir7yksBmVlWb9niAKgiR3CPcWqgHUAC03iPSMALRvJygifVaMh1u7sZ9Eqbbu7eUQexBedu1D2NyzUQU4W4a4CnxSt0x0zBswpTYmctmc2HaEhMGAeBmmoZ3tz7PFKYEAR6utXZTBet6OuMqnjIEBOsG5qveCREJMCAgPFwtQrUlEMQgWitQJmJTXE0BTL4yvLpKmInWGxMetKZhtqJCSMQAXPK02WzMFRA0+vl8AnJby3lwj0CI1hsEsLCwrHDB+vtgoNoCQoSB1gQDI0TB7ODu6ktriXKWkiJBEDqj44qJ17iPaOStt6WfTtaMgiJgsW6BQBLhDECr7DsMzQamoLBwyqI91KCUIU+7nPO4G7qebs/fFBpvjt/klHY7++K3v/34k/f3+4vlPOtGNaykkko+He/z9hGEGpJganVOaQiriGzuA44pFwp27rm0zufD6a1MNJR8qPd90PGSn4wjCWPgzbvbt/f3Ote5xaE6UCYvXWMyH8terfcwQEIJJGx9RndiZgSkTlhK2iTMESQpS8RiCki0Om6ATBJUJGcmFHGnruH4HaHOnBApUMPQkE61IQIRtKa9msiqQgQRcTdV72mUVXZlq8ESI8JWSR4OgdAcBDr0ThEUJOG4+kaSpCQFgwJt9d4GPiQdrDbaImNCPPW2nO97a6r94Z3zQBSLMLDV57mSfADAiUsZANUsLi73T997VnI+H48ZiodXXVqdg3prlThMm3nzUAFHAn6QDkZ4Q1ixdVTtEb60So0AQpMmYkIL7F2PqnNKpEHRAkOSTLTRMhgjmZsqRkiZl7vDYrF0QLXW57aYRygzlSREuGI34fZQGwAgwC6NTGLm2ru2xZysBxJAIDEypbGII9SlzfMiQnnIOdFqgVU1ICLkUgSb1bqw8lpd/kHBJEkIERFac3eXcEKS1aXtEWbwXe1JEOt3ekgVcQcASTKEVsdAErPQtvB6KrvHqjl1CAQiScIB2JdF5FTNarPe1RxozVAIRwRiRoSIcABilJxIZNikadx9/3s/+PAH3+cnTwgiMZJ5a00kMCxJ7HYDsDk2JE3U2QM0DACjI64QCap6bZWYz3XmTAhRmDUMKMzaUm9rOwd4ymksm5QGz2MNOB9P52VGgHHcbLcXavH1N99qf9MsHCgFUO1a56YGm2GzGUUEBQDS6j8GjCwCQa4IIgyEwEgc9KCMUlPAQAimHEFMCQjMw7uulkkgglXKFGHmqusZ63+g2Vf/0tryASCSIMAKmBKAu5krIK2gEkIgQfgqzgpmzjlLKEWspm4JdzXn1a3qD+1OQARiAKoHEaVcyjidT+d1fX09zAEQQYhW7joiRCjllFPOOe92+/eef/ijH/7R848+mrcDKBTOoEDd8sUY5h48TGigKSEyZ1ILsza7o0AQi3YDYHe3cAhyXMMWtPUF7MzYTKvqoc7Hc7X9xfU47SBCkXhQ7pggcTgjCFFXEyBbFlXlVAYClNVd5OTqvXn4wIMgArKGEfFYxiFtBcfMpQf06hbQ1l6KOHHJJeWce3MPt7DzctC5lSGRYECUMiCl3lcRIHBK7uYesdYWyLHCAqpmDsApMWOselaKgAhHAPBVku8ABh6wtm6BBAQovQdhoodK1kUCVuH2g0c+YjVTIToFB6EkIEYSEkF1MP3DSjtiIDogEknJZSjb7fbRo8ePHj15dP1kczEhh0QgAXgLcyGFHGFAzNvdxf3pPg3MjmSVENawFUACchAmZiCSUnJKyDxttlpPAb31E8bsNrd+6lYxVm1pNjOzLPlimNJmMgZv51OdZwq83G2fXF/e3d+rWTfngDJmG1KEW++tt5WAQiIIIJYkOeHgHdsSqqAKTHK526dcShk2291muxvHsbuq27ycX7365vXbVx7NHYgYKeFqfyViRObUe7jDmlSzSoWZGZEibA0PEjAmwgcgM5hWwJeQ0AEgEAC6rSdluLtkKZIEkcy0e41AFvFwDAeGdR8Gydr/Y7gjVTVHJGEmcmb3lYoOc0SH/2B4KLK92H748YePHz0FwPNy8tvgzS4laVVdjZ2+E2KlUjIvKIndUVvPKeWUIsggajcRZhHk5ICljJx1mjYVovf77gq2uC9LO7rjZne13++Zd+HWU1/suOgigUXIgc09Cz9/9kxEXrx4cXNzY11LLiKytFq7InPKpZsD0EPqApeUN0QDeGYYhmkYN9vrqycpjwjCklgSSSKkxCiFWejR48f7F1++ePHV4XRwwLXM7WYeICIspI6h4AiB4ABmLimzCJmZOay9H6xeKiCktUpZD19YXU4RYRZmyJyYpTcNj5QSxJo6ZL7Mq/BemB6QHMluUbU1bdoVmdczlpPoGgIUQCzmbk2FoXUjhu12SokBnDlYklvM8yH6QogCnEhEkpCYKwQdDrdEZBZN+3aYIpCIgchViSiVfLG/AkrmlsrYumYIRO/356o6Hw/LfD/mNGymadqWYWoPzU1aFmiOw7jVdpqXhkS5DNdXl621b1+92l9ePn02nO7b+XzuunR1AJaUADmQhnG6fvr08ZMnm2l7+dFPU5lSypIyoQQiQmLJRMkM1bCkrKE9VISHYfvRx9+rrR2W2UzZIZcBSWttLKkMxSNYZJVJAqKk1FXVjGglK820ZklCFBECKCK1VW+OiXIahKJpX2/ElSoRRFhpjhXDLaW03iMwgNZrOTCCUrfWWkdVjOBcgEjN4oHDot7N3ZklpcTkpQgTuOluu82Jz6fjMEzjOEXQUhcHBJEkyMSEmkqpvZdhcpSqVoYLoQgPQERmKTwxm8fF/noct3NrATCoI8bxeCQpxGPguL8ctdfz4pzj7ni+vW+MJU9ToFzsr8dEkViEBoqcea7LuNv95E/++DzPd/fHnK2+/Da4ba/2V9ePpt3Fdn81braXV48urx+VcQzHBS+AHnJyPDAcSFKgOKYIDiD15AAA2sx89Z6lMk1bs25uqtbVAskBVBUJIcBWfbYkdyekiFB1ImIpIonWkAkAAgAwyVDbol0NlqDQ8EQiZaDV8Lff71rvrdau3T0CMQ/Dmg+xZjm5GaQCHr4aFtV6beZu4aqdUSQlRFL1klISBrBpKizRl+X+9t3TR9eGzYTbAofDwa1DwDRN2yybcSucl+ZELFxABsDIw5QEAaKrRUTKgyQhonFzWUoxkNYagJ7Op1l9mC4umC8vHwn53/zNv9fW9lebkDxshVA48T5dhXXv87jdPX68j77M50PmiXLqqlgKpGEcedo/JknXj588evJs3F6UaRMoBuSBqtZUKcYAcgpCIiZCDqDAFCGB7M5u5BYOZtakgxSMYCJRczOLcPMHQ7KFrRhp710kMUtrjSmpmpoLYpYc2n3FT3NOhCsjEsRVq8NqPAOJGHJKOauqECqCBhhAECOQEAkAelAYdHMzD/RA4pzRrC3zvCxrwlNXNVjxQw4DN2/uJFGGfHV14aHjMLRaD4f7cZymcXs43J0PtxGxmSYOKymlrRDidrtVY6RyeX3pTpv9VlI6n8/LMqeUiOjqai/M81Jb09b0fD7fz8fpYr/bbELbVNKvfvH39wf69NMfXj5+r6k+fbytvb56/Y5hGEq6uBjHTPP927v7wzCkYUiIeJrnSHnYX0/ydBwnKSWCnBi4KLI6LBqqbiiIGWhcOV6kFZwU84jVUAsJUMzAgRzg4SBj8gAz8JX9RQTw1tXCRZC+w+RiveyAmSUCCQ2CwlGDmMhIHNiRAAkApUQkad5aq7raoFfJL5Ecj4c1kC1nIUlISd0jVrnzQzpHU7cIIuFSdEmBqF3VbHWVmimwIJE39bAUqLUVScM4aevz6ZRzTsTWWz2fbt68IJL5mK1VbX1/+QRoGncMvN1Nm6ur54SJLzbjZrNd5tPpyAim/fL6ca+1Lqo92tL70rrao6eXOZUxFVS/v49PP/0Hf/qnP1avL15/1az3aCgBBvN8EhxM7XyeDdGJFvXNbrudNnSe3eDR9gMuqXU7nJdWLUirNU4jyuAJAyiQE2YkeBDuolsAUgJigEQ0IEhrQGCCQISUgHm98lhyBsWIUGu9d3YilqbrxjNQXcXOAUjEKRd3V3PgLDkR4qLWe0tCQBjEHu5AAQScKCzCH3zaOa8eZjL1rr1b93iIoAhkAIYVWQD3Hq5txe7OdQnVnBP4iua1KY1Cea4niDjc3r0b80efvL+ddqfTaZnPrdaIWM6nktM0TUMehpIIgZmI+Xw6D5vNcm7fvni92e6vrx8Nm30uU8ojYbSlEpUkkfNwOp3b0rTb4ydPn7//wXKsu+nixZffqMn3v/fxMF4ezzdlGO9Pd0F1GEs/IiXJpSzzXes95Uwp7XbbMo3H49kQyjidu4KaAyGm4OiOQcmQ3cIcgSiVZBYMlFJiWWUaYI7hBJgSl8Ac6NoWBGTOxA7RW+tmxlkAejzc8pxzksS+Kpow4cM+A49AIkmkuio5eKBEhNoVEAbJLNAdITyMOGWRIZ/PaxAbAMixDms0FhJ28A62tIWSZwZi8HANJXOwjq1y7ykiI05J5gWs6SBljQBz7pgjEXmYMd0cjunt/XsfvO+Jl+43Ny2lYbd5b9ycyzhN26v91ZNhuhoevZ+mqwFK66x5++svv3nz9otnv/vVn/3DP//e939A0yYNY7TuRFaXttjRb4/O+6unjz75fh2KZL893f/+9de8xekyVb2/vXsjlPopps3l7bvXF/vBun39zaupDMAiedxs95zEQHhMhSczw7gTzuC5K/YFLQQ4twrOkMYMgktbkj1K4xaiuLGUQUiAqNfWegsnEU4Fjl4daF7utlNSm7vPlHtAI1S1zgDTOJU8CgIztujVGxAJs615jhiH06lqyzkJRq/VmT0sEBVpqZpLXsU4AqFdJU8IYBHhIcO4WWptrSFRIK4WuAjorgSxwn29NWsVtZP1cF9tbBHh6k5GQIjYm05D3l3vb97dLdXKiPeH4/m3v5M0sIyAnErK0243bXOZ8rgt26vt5ZPnH//g8dOPmouU7ctX7/6f//1f/f0vflXy+Ve/+91nP/rRf/yf/NNn729kHFNOJPn08tu703L95MkHz57Tdl8hJPSuVvW2vZi6tV7bUMrpdBROu81+txkPpzcv37zEgE6UUi5l5JSIE5BgGDF4hEUFRwxhyakIagEZwhGTpHFIOQVFP+04bZDQg8IzoFAwIyYmXq3KtXnYuZ5ECCHevHld+yJJzDoKcgiLiAwB4eaIsloZEZAIAWUFL3MWQGemaD2IOHEE9N4Opto1LWLuZrbGPFNa8TkHCjGP1nrrvQxDziUAyJObW3h4kEhKyc3XtCJwBAQmHsYRzbU2ClqV5iVzbbM0+vDD9w6n4+F4OJ1BNKUE0yYN41DGcZymy+tdyoMDA+Uy7h49eTJdXRXM2uGXv/nV7z7/rWTpQJ9/8+Kv/+7vv725/xf/s//8s8++5wCQpWw3j549e//p48uLizMEuHp3JthfXIwc45DevHoHpqr26SffQ6Lzcv/t5zet+wfP37fehQWQIFBSCYLo3dxZZMi7ZbbWLYgoF+KCNLJkTIlzkpJTFpMtU16DlnIaEDE8mIiAhTiCYNXa67zZbs7nu5cvvtFokkLNVB2RidLaWycKWeN4EYPAVuzNTSQJsMMqETViAAu0sNa7u6wZiWoeLkg5ZdNusQbKruHbOTsiEnezbopE5t5VI0ICiFgk0eDQ0FwBgJhWa39ECCGllPOIoB6LiGy2m3EzsGD3KMMkabi4vLy8fHR19eTy8mrcjCmPS+09RIMpDTBNjOmLX/7uN7/77bTbDNPmb372xek0q/n/8b/5P81B/5tn/+vr660FPnr+3pOnT9G7EZBrdO3LIojXl3tv5/l8Pp/PhLGZpu3Fxe9+9/tvX784npbnz97f7C5ub94BUmuasomqI7TW1VSEGkAIpzSJXKgO3nipkDAnmUgSIEdIHqZ1FcGp5AIR2jpA1wdlTTATmW4nFok3X79Y5pPkqEsPQrVAQqJVgWiQ/4OH7EEiAxDmTMEiQG6qPcC6KjYAoABC2oyjuzdoZgbmlIBJVqoKAGTRGkyJ8yoPYUqwQuOrluu7UAYAWHtHU3Nf89UVbRU5gAjmzBfTvvf2zddf/fgnf7R/dPW7332hbuEGSKkMeZzSMFDepXHjrBbklKvj6NHa8u3rV/fnw9yXv/p//evff/PV6XT6n/7lf/qLf/Pv/tW/+bc/+Qd/9i/+xf8EGFMuCDAfDwTuplqX5Xxo9cyEc6v3t+82m+2j66syDG/fHW7vTxdXjz7+wSf73f725kZKLakwc+tm5yUgmpkDqAZmnHa7cbpWm97d2zKry9BBMERwZEoAFCiBjAgsCEDWGwWgh/fu3i16azVoHoa4v3t1uL1hxgevdhmI3RzNKRCQI0gdg4mQEHzNngoCQmAwQEMGIfbeG6itEcdrFRsWhUUBVdW7UqY/SOLEASlxRKxa9JJL176ahL/7N6Du4Q7fBWAzC/OqpVpzutdMA9fedhcXyPjll18+++C9n/zxT799/daDhnEkyep+WjpkzphoKAgUyFXdg5ARRe6Px1dv33z18pvXN3cI+G///V8vTV/d3P7Vv/l3P/nTP/no/b1atHNFhLn2drwn66q91ZoYh2Hy3cVuu62tCaX709uPvv/ZJ9//JCW4f3fbqodzm+dhHO/v77pZgAcjsQCCY/QIUA2A6eJye3XRWjqfVxWHSJSw4KEkTuiBgezYWnPVsMbgwes7vYj0pqeXL750b8JQ1Ta7LRAj4bJ0VQMQTgysQQAEBGRgoC7EkCSJgMWaoeMU4M4iEOBqqn12R8RhGISlBgjxqlcEDFyDzYgexhgg0tJa7/qHhAlcE4/d17AmREiSmMm/S70j5tVyFeG91/vj7TgOz58/AwAP+/R7n773wQcffPwxp3Rze3talmNtp24GhJLKtG3uXTvnTElO82lezswEDn/yp3/2P/9f/BfMKYJ+/Zvfff7FF+tXlpzUfa611VbrAhF5KMwiKW0uLqkMhnxzf5quHn30/R/vHz8HLtWoKpAMm4vr1h1QENkdmVJOAwDPTY/ndlo6SNrur/aPn1DOediUcZfTJFKECz1wsuhm2mpidq2n453qzBweldhJ7HB4d5zvA2yaxu12S8hu0FuPgJRyEk7CKQtAmJmH/UFhWnLJnJmFSdb8gzVBBXC1Lq+4AayiumEYEDGnLJIImZAl5SEA1Iwl5ZxabdVqziXiIQSCHrRxa4IIMrMC5pwsZ21tzd6SJKsDqC327vYmj0VKefvuXQ/iNL56+9adgdLhOF88zSRi4TmlzcUu57xK6/YXF9vtTnufpjGX6ebtu2++eTlMmydPnkTEty+/7f6nbjAkbgAANo6D1XAzcA8CyeM4bUmEh11T31zsedyps2OyYJSBg3OSXm2zyWrd64KI7qDmQWyBAOyBi1pdGkoet8U7RjdGGobSwcJUu9XjuZ6OoL23ZZiYM87toNjKlu9Px3f37/oa0oXCktZYB3dgAkBxA/NFezNzBiIQQkLmVShkCB4BRGus8h/yliEh0BrHHevVtvovlmVR05UUEkPovbdeycghCFFSyrms2mIIEBbAAk5qagCqWmtdlqXWGuYJGBFNLThEyCPm+XR7ezPt90uz+9Py8ac/VIsvvvxSHT/48JP98VDGIQAGnEpJRFTrgpQ+/uTjf/If/UevXr1GoNNM2vVXP//7P/7JT//0p39srmAmAOaGQb3ObT5LEkqlIHUi7Y0YgjlYyla2w7jZX0keI/T87u39cQ5KlBiJ8zidTye1cEfw1R1FwonTiJwsCC0IA5mFkoe32tZCfTONx8P9u9dvTre3ArApecgoKZZ+anGSgd4dbr598+I4Hw0iIGpTSYmZITDLAwnsqogdH5xyuBqsASivuSZrqDMSCyc0M1sPPIJAw9XUgczA6AjApNXMfDUOSFMl5pyH1UI9lIGIa+0QwZSyJFUlYkRXemA/WmsrMJGIi2TzqFXVbCg45BGTuPvpeOyBc5/vf/7zT773w6fvPb8/LFX74XTz6MllShnA1Zq2Oe12klMq+Z/8o3/89s2Nd99unz199mwYx/c//PDy+voXv/z5lEsm6IRoNh/u+nxWmgQF14QCByCklIdhk4YxOIWMirnX9ubt4ebuNOW8GYbtNHkZ3aJWROQ1/FQiGYVQJkwQiIiSOKdtKLSu7v18ON2+apCYkTLQ5voqEzI6st0db2ocZUOHdvf1zZen5UgiTORqBsgPyfIoa4aiKWpjcWGxeEgetu9UMRquXd2A10RuZogA8DX0D+C7EG0if9C1xkoSRIQIi0YkQE65m7XaiDrFmiYRQykg2Ls1qwTq6mvVU0rxzWZ2r+cZDYi4DCUnmUbYXOzzOLDIbKZLE8Fzrd++ev2TP/7zH/zwyeu3b2/fvX455Ovrx+M4mXZEj3BQq/Ppvefv/Wf//F8+vnz0tz/7bW19bsvx9lYQbJkvt9tQEIj5eN+XZVPSWEZznOezKkgeNtMURJyHsrkwFAdy5nOz43kmKWWcLnbby92Ft4qId/fvluV4nudWKyGmkrU7ePe+FGzIJqzaPdwyIyaq3ZlT5jTkVFjAequn5XxCts1muG23X774vGEr28FMyaJBB7MVqhYkdEDvgIYSQRFrKiUJsRiD+YPpWt3Nw4IwHCF6uPmDtDwgaFU7rAEZ7khEzqa2apxls9nM8+zqksTVWmullMwZAhAfMt5Cw8EfypmUPBcvpc9z/y4uiYiTrBUfCPNmuxmQ+u2dLfrBk2fDdGmupeTPfviDN2++fP3mhWodxvH29g0iEZb8ZIQgTPTBBx88e/r8H/3F3a9+9du//tn/+OLbl19//fXjR4+uLvdWXQrf393aslzsH3VkpyjDsN1uRYiQ1jKwKuRJSPC4wN39WdX3l1e7YZg2m5Tz4XRcYwXXMTHuQUkQxQJaba2eDDecm/uJnF0VvE9jfnSxlWHqS2vnc53PuI4Osb65HI9++Pb1i7v5VsaUWNzJvBsErsngHoZGrq49euMHnWes6ZT0nfdu5fGBKAAcSQHQtZv+/xsECGLNLocIZmJmYYRAVdfehVHAESJKKWWbtFWESIKuiqFCKBJAhThbY5PAQomd2Uom221QIxG7BkrNU9CAkLmhVCUarqYhB2/G3dNcLrvKJBdP3v8Lyq9Ox+Px4Juht/NLMMzCaZj4DBE4bIcPL8anz3/82Ufbv/ubn//i737x5Ory6TgSxTwfFl1wos5qS2MaEROiyLg9nE7V4mK3K0NeZXjzzbeH199Medzv9mNJInJe6mlZmtnSzBSSZGZJTIl81qbuJWWG3pYjjwyYAC1lIQ/KZHOt5zm8BtXFjsrz5qoc29uvXn15e/fGe5+tGU8c7D3Yg8EYjZMQQut1XqqaSxIhYXSRRMwRYL2ZuSQSwDCPsITICL2v9CTjA7loAUbBGOhua/Y1Y0qW1SwQ5XB3eAjvVgew1aIe1t2UCSDYXUWQk0B0V0IU0SRZCk8g0s5LQaJEMGDeE/FYNrs0XRbeSsd5iXFzmcsFxGhKp6PRtM+DtPoWYhAahpwSoWtdzl29cyrtVDd7Rb/Pcfe0+HwxbAtfbgew+vb1SxC4uLia67yhDTpHYuR8OPcaNF5ele3Qe6uncxHcJnt0uRllur6+TMytnu/qbIROjMzgCEbgYaE5RxEWylAKDYUS97Ygo7sF8nk+nw63cAbkqO2+6vHiMl8/neZ+fP36y7d333ara/NJDcEhAQuEMDIGeDcID3ciRXLKBpS6IidCCXBEZ7LEAgCKgQQJHWx1fiMxIZJZi/BuHRQjfCUDW3cmYZYyDJKSJEQkQgLXtuaKrDUuCSCiBXYDSWzmvaq2jrpor7Uu0WboSu40jNtxiqQNeyBbIDsC02azffz0itNoxu5R1djclxOFPX7yeMzkEZwSkiTJCmgWkmmZl3a60ePpeLg3XSLqMt+8ffHbPO/O59P2crcbdg1YT8osnJOSIeHVdpuG5BGcKKWBwbe8v9hdiowRPt/dns5zt2DJNMIudlZzrYu2ygwQlvOoVDoW4lTK5nLc9epv7169vb07392Rx/XuYpPHwDoMeHW9dW+//dUvvnn94thmJyJiZCIMQiBEBoJwc++9eTinNAwFLcwxwnU145ghoEUgkq9ZkMQsyQFaa8w5Yp3UEADoiGFgraNEzpkQXL1DXzN1RUR2U3nAWIeBZY3jde0WjsAUxDIk5tA+QwSLYLAR7Pe75WBvb9/5stS5HG7fTdcXl+89KtMuT5fAU4PcDJb7U5l4t73cby/GYUTOhhnD3928OXJcfP+T/eWVuQHLkMtpqe6xzMt8+3q5u337xdd3L1+d797t5Prbr351Gc8xlWiDnTuZAPl4Mcxd75fl0bPnqaQKYeEUnhA9IOeEzr26a63mxtTDq/aM6EiKoOGYOOchwpfutVkIgARx3++yzsf5eH8+3KLr1eX++mo0ayNBoH3xxS9+/8UvX717beSUcxoGSqwQjMEU+CAD7qoVwjklJAbkVe0WQQHkDuvYpFXwHV3XYWEiOSIATRB9naS0jmQgzjz23tCCAjLnYAddBwkRAIj1BQM3wzBMY1e7PxxKHpHWmSECiCKCNrs6AuaUPLoRWuiynM36tBkvd9tNKSb5uFiLCu0YbMGbNFwM290alAOSe6C1tt1ue/fXr19vN8O03e4vr+4Ph/NSd3k081bb6XRsd4evf/v7m6+/fn6xff+HH6TNUHKb798cFRPmvHncqpchH5fZEK+uH0mWDms0IlePWhsTVIXCMA7FlEaSPAwppeO91PnY3ZqaYTAiJjHlqktTSjmxZAw83R9CVRiEHNAEtcddRK16+OLzX//y13/b+pzHlKaBAkLJCLoFhjMnCEcMZkSUlRoGZrM1sB2BAFZZ6INHPCKiqzMLroMsCVMKwXVyRtDDNhVhYiB3o0ACFM7qD+GmACAcQEyM0GuttfWmOUEgSs6S0zK3XrWAWzfwIKZu2mqrx2PTPm2GzHSaj3U+jpdPxv2TMm4gjYpZYUAparG0WvUeOD+6fjyUfPPujXXd7bcfffjhxcUVSkrD8PU3Lz+gMm4247hZaofg6+2+PJqvBsFo9XRzbgfa7jtNtsw6q559bksT3OyvhnEKhFOD2/vD119//bO/+7sXX319td9eXewHkQ+ePvvsBz+42u9IUt7udoJ0C60vSJA6W29NNbDkcccuQMUNA2M+nyi8L+fj4V09HrwfP9xtzOvXX//m17/669PhZhyy17n5rMjGabq4lDIyMlh4WKxezzWNNpDWGVj0EPBmAGuQ5UOgZgSYpZzd10IlpULc6xpRuy52ILWmzJyTBERbVMnxoYdAAJDdduMOh9OxqxHLOIwASGuiSLCkvHRFIIgggDBvSz0fT/PxNEi6uNiRWZ2XISVKeZ67FN5t95wvFKS6dKdhHMzx9u7Qmk7TZL3Xpb7//vtPnj7llPePn0613h7Op9NchpFYtpvt5vrZFPjVu5u7t2/qfDfsSk+8Habdfuzn+cXnX0ZPdL179PEHeRxM4cW3dz/79e//u//h/zNOm8Ph8OLLb//9zd+dj8cXX33+o88+/su//Kf/2b/4Tz/96MNQ80AUQWJicVunEiFQQkC18NaW5qEHV/O+3Lx+eT7eJooyckr95t3X33z198fbl4xajwdkTMPoLFSmhHYxJavmrXtEV1u92oAsOTGJMDgggJKTPaS340N8ympZpFXJqBEhTIIrJuC0dpbgtc15EEY2NUQEQ86MhA+cxooVuUNKee2c6jxLLkI0DNPlOC61TjDf6DzPej6fWtdhnC425fpiW5iW0zlUd+MGyuZepQzTbns57h4FDyED54koLc3O59ndkbC3SozPnj17+vw5pnR3nBnx408+e/XtN3d396UMbhbdbdF2rIXLsL1Ege1uh5ihx9zPoCE4The7durC5eWLV//6b37+t7/8dV30vWdPf/erb372t7/59d//nLs3qIHWLd6+efsv//k/++mPfrDfbKzVlIv1qr1HQE5JneZ5vr0/QiSmvJzn7TgdD7fufX+1vbraPrq69Pjq5Ve/ffv6y3p+Z8t5GNLFxYWgYk4yCIPqctAKDOu8OUfMyGQOGo4esDpBmSWRB9p3+c2IuJodv4tSQTMjWq1iEBixjrAA2OwuhmGEAHdf0fBu8x9+V05LG4FYShlH9yCPkkfJKecyDDklNuU+zxFWSipycX0xDYJ9Prb59ObN3e2bt/PpnIhl2sG4u9g/xrSpJmWDlGipXsZNkZIuLlQtVHmjtdd5OUlOZdi8eXtjau+/996jx89aXZa51qVtuQhmxqKLlTxg8LZcw7hRKnMzbbN5H9rll7/9zbTb/+v/79/enObDzd2zDz8eeHj94vVXn3/9n//L/+Vf/9t/9/nnv7y/Pz958vyrr17+1V/9281QfvrDzxBJODEyE4Wi9l7tFGGMEWFZYNjvCGK3Gy8vh+ur7cXFKIyvvvj5m2+/Prx9k9FIcKCY0MNaNOKUYon7w23iUaYdgiRGFHJOEYyrwACR4WGQT6gSOMZqykAhSCUNJRFRFnIPZoyqrbeHPPeu0zTmPJRSmHmV4YeHLe0hkwZA8rBzRMfOUrKQcCqlqGpttZ6PYAXMUuKcqNUw1R66HOd3r18f390c7u+X07nPPZHw0CAfzucOlB9TmbbXm+2mOwPSmhl+Os/nw/3tu7fDNLXe3757t9nYdn95Ps6v3tw8e/z4YhjOh8PV1SX3/osXb5dTXI5XA2OeRrfsNWtGVe2thflFP738/IvqIYGffPDkeD78/lc/B+sDRyxnsr4ZMkQ/HM6pjLvLq/m8/P3PfzlJ2o2sSwWPLKWwBHg3HQpD5DprWCMOEd4/vd5dTJttbu309t2bF199cffmDZtNhUNjRLDjvRFiGVtbPBUXkQHdMiMQCq5JkFI4ZQ9a3S6rKztMiXFNykQEBJTEjACwDje0sFW05kkYIrQricg655MFAlrTrnY+nSN0Rcwlj1uAyCNeXOy0t/BYliUi1oHfob0vy9u3X797+9JdS+Kri62UAYDVKPEom7FTJ+THz55MVxsHTpJMdZnPedjkcUNJiJJlS0KbQRitWVxdX0+b7c3d3dLj+fP3Wl1u7+6vdpvLR9do9u7tu1/94jcTyidPPxkkY+YZzB1Dvfdlng+9t+1xIruzpj/60U8vn3/wyfc/+O/+1b85no/bEZ5eb/5v/9f/MwZOlxdPnr3PIDkNP/7RDz9678l8OmMn9g5qBMDEpk6oTJg35QzUmjHH4yfXu/0G2M/19O7+5uWrl199/oXXts1loHBAMcewMmTHONWzah/3FxTNdUEiJ+R1Esca/7yyswCq6qbMuLK2AE5IRBSural7rKZUwABw4HWiSjh5d0sydXMCj4i5tt672TqfJNxdtheXvXdESGVs3cK7fjfFpetyqvPNzbv59DoxXl1eI0TOAl23my0akFGiolW309apf/P6y/PSIb2m/BXmX07766fvfXD16On+8ipJRlwpfXp3d/v61av9o2ci+bws81Kv95f3794tS7vY7+tSAwmQzqd+POrzTz+NxGzLmXvXY+u96QLRrd8NqW52m+dPxqun2x9cP0Wy//5/+NeJ2vPnF97fG1I+9vjkk+8R8uXl1U9++ifXu+nFF7899zoIQLeMDoKuBqHgMIzDNEyumMo4jhMSHefTYb6/ub/96tWL491pzCMh5eh5nKAvqp2MQEkAzMHr7CAoCSU5cJgbGKAxreOMhQnDTF3RnZDWaILVpdpaM9XV/i8sRNhAh3EEJncl4aU1oLnWtnLI87yY+ZQYMVTVIyRtdlEXbXVpzb0TKHlzbcv5tCwnU7W2DEnGIWfJvTdA3lzsVsHx3c3d62+/efPqLQFePd7LhreX+0fP3798/H7eXHLZbvdXV9dPJefampmbdg0OlM+/+KpsH33w0SfTMN0djymXzdUlWL87H3Pmy2fvf/TR97782W/ffXMDz3G7uWLo7Idznd0wSFAomBxif7EdxzJsiun5L/7iT0vJ6OZat2PZDKVDevLRxz/94ff+k//4H//ks08O714X1OPtGxpkWJtBDySYMrnBWu4ZIoXP7czu2o7ejsv967df/x7m+xIa1sy6gmlt2hU9IAfkJJm8VndiYEDBRIgVAc3DNUiSDBNxBuTwHu27KIY1F8+gtdZqM7ecs7A8jI4keZD/r4Ph1ogWUGJZrVARThBEwAHyrvfNODLHcrrDfqyndxmNbInlrt7fBvg0jRfDc6aREqV9MfTb0+lurkfVt/V4u9x0PgvEjHK1f//qyXsXj57JdGEyeio6bE6Y5mM39VJGKZvrZ4mHY/c416gtgIGH6XZZfEzDfpytmbX3QIbhYl92O+X7n/02vbeMz64k88EK+lg5aEpHHGlTPF9G3tXFeEjQ2g8/+37J5fmzZ198/sUy12Ecvve9j//kJz/6+P0nw7hsep2H07v73x8OJE8eO6TaVRJjPQzT47LZKGx7jxotp2b9bjm+uP/6y/vf/i6/+prwWBKWadCF5uNRxs2w2bWuusarORFBO59t6dGalAVLl2FrXFoskUYngiRIIpKjU61tjTpX894VQBwMkIEGhwSBedzUbsKELqEmzAwoiFV7EFBCM3erzDjkBCAySHg7ne9uluOteGXTc5t7Pc3n2ZyHYSplGjb73sMJgUW1iaT9/rIvs3elwP3FxXYou+tH26tHQy7adGn3DRaj4/Fsj5/idn+1u94Pwxhud2+6gbfW3r55I1Iurx9vtjvOwnzW1pAgZ7FeD8fTqdXraXsiX968jPntsqFDtkj0aP94/+RyW+j+7o6EzNUh6jyT+u5i/4PPPv3gg/dPh/O8VGLe73clS6IAcUfGlDiXu9t3nWK32yOTdYXQDMgppTRS83nptzdvX371i9uXn5/efDvf3jHi5fU1g2dJMFpmNrXee2hfcwsxYGkNgMDhcLjHqmkEsohkmDZSIDGim7l5r6DWugZYzpk5I5KaTdMUQIgoLJwkhHV148OD+rDWGgHCbA+6jVAL/G7ktSy3L8O0zUerp9AW6G0+uzmnsUw5lwEIZ8ele2bJqYA7JSfXYRgeXV9NguQ25Xw+n97+7nSxv949fn71+IOnTz/Mm6vgYZx202Yvkls1s8aS1MwcuurxeMppFM7cNJqdmHLmi4udGRjS2ezl6X42G7bTUpfX9/e24Q9//MlH3/+Ik2wnvrzaV7VcCjFihHlvvZY0ppSuH18SgQcEQBi0eorTPJ/PTkw5Hdu53jcuKZfiqgaWmqbWR3aA0La8+OqLX/7sf6x33+4SDgl3uw0hqC4WHhiKULWnxEXGzDSMIwjPdTneHVqtTsQOLCmIGFlyIe9ez6baPcANH4ayICAhAhBBxDgMgay9M3FKefFuZkycWJAgiUSEIGi49dZaW2fqEBIaAoC0wyuKkLDwZVnOzQzcRIZhGDmP7tC7RS4wEpWEaSD3PrfWWkrp+nJ/05ably/fHO5Aynj9JHHeDtvNuM1chjSk6QJ5zGlIqYAQ8cYm2Z/Oh/t5HKecSl3a159/vZl2l5eXWaRaW277KOSSbRjenFot6b3rZ7vn10xdNunx80f7q6ssPGyYCrfaau8W7gHde9dWykAEYdGb177i936el9s3Lw+3r5bjbTVrpkub05gmHy2C8+ZU1Q8n89HdD3e39XxAq1PBx7uJeo9e53pW68KSgCinQdjdOGJV7zXtc61qDRESE1OQd4xWxIcMhmq6uCkRr0FtOWdY55e760MikRCiEQEREGXO8DBWmb2b9o5EsM7OBmDmcCSWkksqmZAk+3kdcqp10VaFeJg2SQbkYsYOkrJ43uShAPjSdc2pi/C+LK9evnz7zde6nLfDuL16vH387PGT9/bXT/K4gcBWFcQ5r6qRJCnlLM396snTw+mr81I3G9Bub769OY8NlS+2FxBgs/7d269e3tym/eWT7z959PTxow+f5+tduR6HUXpboC055dYroJt5M5OUXbWr1t7scAcOzCLEZjAvvbfz/e3r19++PLx7BbZouCEelzPcRuA+lTEoKVJXa731urx7+/L47s12YGaJPvd5Xk7ndDHmxAjYezeKVDJ9N03oXJdlWWpvZRjGlB2gNtPlmJjQRuizJEABWwN0IZuv4X7g4bFq3ZBjlY0S4zruIqcwJ0QG1Nq197S6xsPgO+5Q1buDAKOIsNZ6PnVVNRemzXa73V5FpKVF657ysNs/ujVz5Fpr9Pn6YhCm13fv7l+/vHv3DtzHMhARuIfau5ubU/W8OcvmKk9aakw7nDZXNAhRQmKWYX/9GL54MZ+qiOw2V9vxcjks9VRv5lsMZJR6Op7Unj9/74/+/B9ePnvskzQJH6hn4ESIUJf68u03izeHoFwePX3SzO7uD2lepmkjJIm5Oi7NDnO7v3u3zIemRqWQQ0DnkvudLm0J2OUinhIlQWbE0LYc726Od2/3IybimOdELkO+m2cgXNMuOaUyTcIcEbXWakpJrjbTJMIAS2vzfLs0ZeF24N563l3KAA/zqzEchzXvLRBzLpIyfCeTYklEHECre4JFBEmJesQqRm3aUBiJ1I2NVaN39+hy++ZtAHCScZjW8M0QNmVHoMycR4BUxtK8q3aGmOfj25ffvHvzys7nej6G9iQSDsfj8e39cbq4vnwcV+ViFFHtxzdvj2frRh9+mC6vNmraAz2QU3HsVfsF0e5i2m+v0Lgv/XQ4n8/L9tHTCgTT5p2qtTrsuKewcJjbCHaRMgdeP33vUI93x6MG3B/PBtjUulakVNgX89760tppqcsya6/ImEsJxwAtw0QiAOvsCpCUzSzIXXubj9aXwtHnI8a5IAR4awsSiqRhHFNKHmFm1RQgFByJkiQhrq2DqZoRRhYk0NClWW9mZecy7pxEHULSup2YuQxjKcU8amvMMkybiDidT8CRRACgtda1r2Nny1CEZW6VmEl4GAdXM28UWbSDpMRcOA+SC6YClJ3JKEyhNZ+pnbs2X8ZMyenbb3737Re/Od/eJFd0g/BW29Itj9MwjWOSxEjoQri73G0un1MaD3P/5sU3EXj19PGpLqfjcnX9BIJvD/fTuNsMEzICIibZXu7314/+f1T9yY9lWZKnicl0hnvfoKo2+JARWVlV2V0NkCsC3PSO/z9AgBuCRLFryKyMCHc3t0H1vXfvGWToxTWPbtrCANeNwu1OR0R+8n2/vP1rlNVzbQHJY7/N8q6QgE3cjgjT19d0Ltd3z5eXj9Ptsbf76+uXr7c59VS3nBIDhrn67LO7GREQYFdv2973HUjC8fXbrUpiym7f3j1/rJlttMfb1+3tS9tvOL5Nf4Rw4VSWAkgpZ4i43+8OUUrpo5uZCK/LMlq/vb2aQclCiClnNhdGYmdOmJkIhWlZVsX6mAwANedcFkT8nseJHIBzqocj0noqNnTOMXs3s4MfX0plNwVXs97HIpkZECJ8SilPda2y1LxUrkWDLCiQyAMdENAjsgBoaN/7/rq/fu6Pt7nfknBNHEiza3xn4Prj/vr59TH++19w+c8f//wf/+//6//jP/6nf/y5nLe951yRaFkuNuG0XoTk06dPX29fVfW8XAsvHsiUOdP6/qOc2mzt9bHX55eF0+PWWbFkTLlKhLf2dt+V5d3793U5hZRtGr6+bW+Pt29v4JE5lZQBRx83VZOc1mVlopxLe9y//v56v/VFJFHy7pvf1lxpou3j2+ffbt9+j75F3927ZKGEFGQ2daqFjzmDySNKzikl/J50MziQnciJgoQ9TOfQPrDYZb2mJAbReoMkKdUIQJKIOPKoJIlY4hhpwXeTxtEUOKKjlGiMcVQdx1aFAIArQriaq8l6/ricVxSBTJizzbm3HsERBETHJrtrz2T37bW9fVozpdBH26Jk7WoabhhBamr7SPWUc1rW8+X9D3mtv/7y148//ePP/+7988s7YrFA13h6LqPvp+uThd1vt9t2y7Uuy1kwoVFQlPWJ+0Jv96HRtiZF0srtsX/97VtG/eFyfvrwQfY53V5vTboD4+i+1It8qNv9Nluzrrfb1sfX+/2vt9tel+XHH//h/fv3l9MFzW7r11P5msCXtL5/fk8xR9/mrccwnz0xTjBiEBYgGmY2JhOrWTCezifO2dwOdWl/bPu2o/m761MsqyOBT+07Is4xiL8DxM3UyAIFPEKIEOece2vEIik/tkaEdnRciQQEhvt390mEuYOnnBDQ/OCOZGRKcyPwPkZru1A6kywz3C3YcFfXAERCJjqAoQxjvzFO8ZnQ3Sf4SByM0XWGMXOKI65Fhui1luu7l6cfP14//On6/qfHdvvll7/9+NM/vHv3EUhAKcL76ICQa6G25ZSdXWEmZhIJhMlEuVxfsu/t/vWt9379cIHsiUQIxpxfe6vrU6Z1b7sNK2s5X548kKgvZU2INvX2et/uIPht9GE2W98fj0ffm/ZxPV/H5eXt90/31+3DO6PkoaNvU7ehY0uM0+04cCBROFpgioMFzIlFJE3D3nuomVpCKrVclpOvpxYx9wfSEBEECEQn3LYNnOWcSzkZsw49/LsHTDUAI3xMPyQAEHAkalxVVUMVzCFgrYuBo8IRjIoIATMbr9++fPr8qyAVwOLe1AIDASiVRQ51sQYGJ5aiMvsmjCHybXssOeGyWB9hccDMhoIkqQtzXi/P148/fXz3Dz9e3v+Yzy/3Zre317KsxIk4T1jMJwLUWuoiT8/Xx/3hHo/9nouvi+Scc60xtIaMaW+vNyQcewsLxYlkA0N99vsDOU/V70OcJE9Pz3OOvjdUFxK4EPjXxw1qZuRckrj73ke73ea+LXWVl3fo9um3z1vuSz6Pbd4+v+rjzfrbnB19ArkzCyfJkBEzgSG42RwDhY9/ymVZlksWRO/z8di28H5/s74fUd599Ee7YRkFJJezbpsTYBKAKLWupZq7OdRaWx9EBoiOBHEoVT3cCSmVdCwFBAIxCQoxDVPt2+z759//9m//+i8yl5iw9dlqXdo+GRmUAoMQInDq2Ns+AHg5+76p3cnLvIPewyegIjAC0fmyxHKJdx8//PwPHz7+eLq8pHqeA2DbMtf98fbX/3Ifr68ff/4TJmEIxmV9ek9Cb29vlquajTFu25h+XyGWMGaOZOmlnM7vxrTf+yMZssRwHU1PawnG+/Z2vVz3bfv0y9enp+fr5UJSDIa5q/pjzN4b9xs+bkHlcv3xwqe9D/e+tweJ0Pk0fb9x2MyjbWST6+xt2+fNTQm4lIWWMmwYOdcc3/PkBmE0sZRyXs9zztfXm5sjgk2qkp6fPmD4tm33+w0cSkbkTHPA/RumHXMnyZVXBkUbidOhvTjX5dgfBUDTKWZd0YwU0TCzcID7bHM8wHsVSD7t118fnz7Zr7+8e/smfNi1RQDJ5nSPnMpxEj62qiLANNwNpx5U8WnWew8zJCpLredzLjU9vT/9+PP5+sySzSKAALEPRZ5jWuutj798u+0f/kSXy7mWAggekUv5cFojYqoeS4l7bzFGHGXZsrIkAUBiwLA5tG89TEfnNbc5wmfJJcLut1cwTcKCqGD7fu+tm+kw33rvrdf8e4QcW5d1WbZ+W5ZaSN7ubwBsU2EO732OcaDdVX1vMw6oH6NqiEhZapJEAQCgXefUMabDdyosoSQu5j57670D0flydcD7GL336ZBrMNewUDONyYECaEFuhjhVXURKTiwCDsAO5u4eNmMYE2AcOeh5e3v78uUTffq6ffu6329ZWOAPwynA9+YApGMUiABIxBDo5qaddB6Mh4Cgwzkpcjqfr+/e1dMpXT/W55enl5fr5V1eVuQKlIELSDk/VU5ZJFOqy/mUl0rC042QUi2S5HB4RcTj8bjdbtDG3kdXdUJi6aMRE4XZHK4qfLjt9LKUtjVkPtXy9vpqfS+lntcTuPoc2puOqY4O3Gb//O2VZV3WS14XLgyPQP4uKuE/CL1EzMgIpOZh7s5mIDnXWk/rSQ7+nvmcc45+5Eh1moiUUhAgUXXD7XHrY2hYQOzbo6sbMXFmIjqknwgWh7jRgNEDjpWvICAWlhRh294Uwg/YLQCGSwBCgM7t9nb/8tvrp1/h85v3LkSnWgWZdOiYlg99cZLj1xy86QA4yqPJGj6QyMzULCAcQoik5PVyen73Xi7vfTmpIzCX9eyRurpOHZtKjvO11LKmZaGcAAGTJCJHREQHiPAEnGq6pGteKo352LYxJ6fsiN/lBzbc5vf5OELbHpfTh+56//bl/fv3O0LfHqjzMTszJ0YOm+acl9OVSAZi7uoVDpOuXJ9f3u5fHntPudy+PhgtEzIKMgMicxLhda1HHEYkmxfQg1SJuZScF3BrrQF8l8MCeG/jftv2/VFLvl6vqvr12zezCYCUUIRyYhDSw1QE4EgOaEgBkFJNGYUwEHSqixz4MZ8jpnloyhxjv3365be//Eu7ffPRcc4l56VKAhAIwu9B8XSsCgMQwh/wMKIIyKkEjphiRB7h4VJyIkLi6f56vzfz8W3veKp1eXuMHyYs61NXvz8mcj5RRU51XXNdNzACovDex5hTkrx7987cFEyHIxLnlBACluQZSQJAmKb22UGb2hg62mN2jT5aSRR97Ojz48v1yxft+2Nst5qLJEnknhPi84V0vaBOZ0hOvPeh2kvhw3zFzJfTBUMTmYQIagbd7l9z4mVZRMQ9AJOkM5N898GGA1qEIiXTgRC9t6mTMaecA6JkSYlV9Xw+L6fTfW9dBwzKOdHxcByWRyIDVIcIkENz7eFuY8630Y69fWFgQnaK7d5un1//+pfHp99QR8aAiAzIDuZDtrYjEhAD0RjT2jidTof3BpEQncC3cdu3jaYik+R8ulza27c+R11WFBo+99urJrx8/PCnP//jhx9+znVVD0n5x58/5HJezk+lnoGozTnRzKZt99776XL+8eNLSmxBY2gfAw/FM2NeUsIcgOYgicxoJhioA3S2OUYrGe7fvizLQqFfP/3yw8cfquA2u47hs9daEgKWAnQxHwVZp5s5Mph617Fte8okkr58/vJUzr31XTeB4fOhc84xCcJzoFCStCzrur4TPm4N6+3R90EY5kCSAVxb84hwC0QW6WNs+wB3Dx9mEXpQht2GzRY+MTIxOYB6qAexWES4EWMWhMQZBCPQNaZa30HH11/+8vbpl7df/yY+Tzn5HBZus0/AnFg8ACJYEhKXInbcfOHgxAzEwkQ0SVKiMCfKJTdhSpxzSrXkWk4vL5TShHO9XALg/rgXi7o+saQxJuDA1oPSKqksBZm/fvniEcu6rMtKmFQ98NCyHJLCQI7DVjumAkAqXKJOdNAWyqicKQWM1y9f5P17DH/7dnt5ui61JIYgYPC3r1/Pp1NZl8fcc1kAkOiIGpnqsDnMdW5zjp5YHvdHWH/cv1l7ExgEvdSMAH0MwLSuGSixZBaBI1GPgpQCLC/nJBjhkjNAaLd23/b2IEcIGnMgYUkSAODBhDkJ1TwJAzRC3CzQkfMRdHq6rGzTxl5LOkkGM2sTKYb2r3/7y1/+2/9vvH6Lvp9KghGhEy1IQDjlIpJSVrUIMIuURFIquSJxROj03vsYExhzzhBqB8g254Y43cIsC62Xc15P25TW2+31tdb1dFqJ2cxSrkQojEspS8kktPUuQiJSSiKCOScePl0hlhQBxMgRbhHgksjMRtsR3HUePjdEdzdiAPDWNp2zj/319evz9enperm/3QIAPI+2B9haz7330ft3kIQDU5REmOvoYL0nZqnClItYf6gNm8Ml5SwSQWNO2PcABtqEj23XCARJGdBzImJ0HXggDt0O5QUQCedjbQkQcxZQV9O23QgF+BKhnsIwHJgT55wYYvQN5xQfFnN73Mh9tk232+Pzp9//8j/m2yvqyMwJ6dhfYwJCYUJEEGJBh3D/7t1FmmpEh5QRJWXidB+bjgFjuKlDsAgyIdP56fL08pKWCkKLrO/eva/LCgi3b18v1/enywuycMrX0+l8Xollb63tLaeUc5Gje4jkEMKcEjpAHM5qwPA/4vcO6koRTC4iLqRhvbdSmVnGVPCIgC9fXjEoJ/FAM82lbvP+eOzlvPp3OduYczJCFgqGYyZrwo6RUjJtqn3OjqDMMbURVZbMlCRnqRXwKBAALRAcwkXIA8NiDNv2oXO0fQOby7LkzDr6bbbWWwRKSUmYD6GeG8xGhBgFGIjggLtn4nb/lmzkwrP1NNts+/b189vvv95+/3T/8ju7JsT03SLl3Wwl4gS5pnoucjitU0oiCQBMnQhV3aybBQAwcpKEntruvbXHY3vcb+q2ruv5csklT/c1J+Hct/vb27elXl4+/ABhoWNZ1rSU07qURPu+399uHsg5gRsGZcGSxcwTYxIwg+kBHkYadNQLAwCXdRn7FuZEB8ufhdHcOSUMyEv1gLbtX75+Pa2rhfc+AFByHq3t+56Ya04zNHxo32+PMfYNwA8jYk756d3L6Dh7IkvMCwGHjgAQolyqpMXM1jMhUpj5d/KakyQidjdzMIOpsJ7OYeP+9jq+3kMVw3LOkgQQ1eIAu5N1skYhRE4MIRwEGF5yUnRhTxh7e9C+bZ9///bLL7evn3R7VHZ3Z6Bw7w4BSHnJGUopda11LQKITPxdomuu8N29Gt9BU0DMQ62PceyCpJxOp5NnZqZH2+9TI92e/cPTKQuvl8vz6XRdz2upJaXEhJkJQtv2uN8fbWtyfiIAMEMi9LAxiZmPSZ2Cq6lZZDtUnGZGRJLSfp/adtRpoyNEyblbP8bfxJxS2mFXMzXPKavqnEpMAKFjlrWmlAQ1JsaEPvd9e9M5MICJKPzz5085QU4MlU119FYShTsLns6ryDpnOBhTHLSF8AhnCzd1G7P1efD65hg2u3sQ0neU6t9l4GEQIEjmNtuDWDAvKAahERSux7Cy7/f+ut+//f7tL3/7+vXzeNzJZ4IoSdocTAREZgCcUi5LiVwJKaabHL35w1x3kPvamHzoGgHtYHx8RzmoRZBwLsUppk5mfnr/Lp/O6XS6rOdzekqlplxrKctaOZcAcNX9cZ8GfUwEyHJQDBwB5pg6385PV5uEhGbmqhCRRA5hTE7sAK11D3ePse19u0XfwC2SlFz2fXbQMcwCEsmclnJd6mnf9tHHY9u3/RGuS80cnpOQJ90pMcaEP8ZCZmjBiBAsnDgnVteRck45Hb4KIjIfiCSS4YhbuJu5mc8DGuOYUoEAEVyWaqPs91vfb2aKjMx8DJbNItgJc7genpKDf9x6G7vuX7/cfv/r4+un7e3L/Pp1tP1Sy+W0+Gg+R6iyZJRkCEGCqUoeTOgAbiY5r63tvU8RKaUuS+r9m3vUKkR8v91utxtmzunktPXe5utnHveMhuYx2wimQaJlH3usaQGYc+77vu33lBcp63p9iZHaBJK0nq5LFbMJEYx5tJ7rQkDfX1IwDKaHm17dk8EknhQzohN3kAfWltlHju2x82QOTaA4NYNO77Z3ldTQJGVFH8Q5xfb1b58f+OOP/7Be3w1Ad1lPvG/e2zcpURK4j8d+27bJMCkaYheCcJCaS1mQxdwDgD0hCnkmooBAMIhOhJQZA81VuAtMm/3xuG/37fBJM1cE3LdN1XKpJckEDMTCUHCG3ufcUGM8Htvt29fffrt//rU/bjFn9fO1vMsi3mZrYAZBBVPhxKeKpRYpso22R+TAhCxmzpzWNX/vMHksy0JE67qKiOrsoxu4gwIgARABYeho4MFMPqY1NdIGY8K2t6amBLSeTs/P71fE7Y7BdYbE7I60Xt67awAFhhkcTokwCHcLVZ+mE9qZmQQIAyMU0WZ08EaouUiSFdyjBwcGhI5GEIngQLrOgeY2po3pBf2y0Jdv2/Z4W9drThVDPMH58oIQvb+az5RQpttU5D/gFARTndoE7CmxJE4p17QyizAfjsIAdSYiiJCdbdvbHDuggdmhF5TEAhm+E6EJwgkIA/Hgf7nC7DG0t75v++317cuvv7x++cw2c6KUBHsqlF29tXFYjS7X9XxdWSjAgEB1K6cFGIUolSyHc4wP9QKAux0fxUPleezVWcQ0nxrAOZXFretoiMipSFlyXVKtUhaQhBCFa0qplnXMqW9vJ2CnmM7L+bosS9cZcEDOGMAADuCjAoajHefJaDcpOWV0bTYbglJT2AzGDCSCyEoKhEJoNs0JjJlYMBwAnMIwJphZjFLW5+caAbN34XLIgXOW/O5pu6trv5yqzzliYihGCGEWhsPxODXIpNBaU8kFkQ9hPYA74GxuphDmHhBg5kzATCIp52BmcJhjBMayrgfsEgJwDtV2fxtwu6vR/dEe932739v2KJTX5ZQIfFo4WUxzDfRUJZf09HLJJTmYxbEHyel0OuSO4XHEEr9j4v6OQlVV5u/ywCRJ/cCaV1lOEW1qrzmVVFOuUp9TfaK0UC7r5XpaK0Rsj8d9e7Dk6/NqZr3vM+R0eco5qUMSkZSI6Yg/I4JZAPofDkuE9iAwCXKf0Qf44KHU1XZ1RHCwps09L0sqdQkHGz4ngHo4eIhwJcAM3p1JLufSx+E7dyAP1/vjbUm01sX1YAbN0Bkx+DAGEp+WhYCBEgkJBYObKaJHyCFeI6SU0pza9vbY79t2n6Pp2AkDApg550KCBwsxp4SIScRUfXNte3u0fR9t+NZmb5OB363XNZcwnW2fHibu7gaKGXOWsuRAGDbMp+R0uiw5Z0Oexx4ggBywqXVdU0pzzm3bjut3gI8O3iIp5rquCaCyphgYb19/GzY/Xj++fPyxrC/AxZDzclpPp0NYlMuSUp1mf/3rL8v5+tM//ofL5dJbL9eXXErOGSmpHbZXM5uBxoAIjod/cW4BxGFuc4zde4eh6MDIGCjOaobhNdUkJ5jStlvf5+wdATiMkNiNCTmSebj5HK3U5bRWDrN5n2MjCne/32+CYeTggREUQRHyXWOD4WazDwTOTJwiQigR0QHZVD2UzphzIcSak9AfogyHNntrPQnTIZuGo6LjPKJSdB26Ne/GhiUJu/f7o297eLBIj+bheBDFBYN8H1tZcl7K6bTWtWo4IGMAJYgIOT6Hf18zL6XE9wOVHew+dxvDJeWlCnBs4w55q+dnDJzI96aWtHABkunRppacTpcnVWttPLYNkU6n09P5fDqfphMLl1KOcPQYDdF0tDZ2AM2ZAUy1o9nWJlHkxHPsj/u9bQ9VZZTTmoRS0jjVgoWPd1piLEmsxdDhYS4ASKGaU0pU25g50d5a77+WUkrJ//APP37+/Zf+uNVSGM2GCho4myLYBAcWiQA3CwQnDU8eHqYRAR6ITIhjDFVNKef8cizaEjkj6tR930dvvQ83AxF1G/ugRhGgY2fTy1JAde7NfLpT9LZtbU4Dw5KLIBsoMSWRVFMSOSgLlFhKzmtN64KqBgXYOTkSyZGSOzjgKaVaKwAQUWvt4KOWUu1waPtQjemoQYYpwu9tdH97kuX0/L7Uc5DkUvi7RHgQ6/ly4bSs53NrnXNbzlfOwowROoeFqY6uNtVagPo8VBUz9FgZbuGKFG6ze1OdhIITS148ATFEWIRGmMZMQkzAFAiE4ctS9qazdZBMyOtS1Px2u9/evrnrDx+eT6VyzNm2pdRUYc/yePsWk5hSeNgwIDrsoUyQswTjVDPV4CyCHkhEKaUZbodY2sx0YvgRdyJO6+WyLEuEt3030yRyVG0cQQQ1pXMtPnTOsOlb7xF8Pl1zKtM0Fal1yTXRQaZmIJZUk+TkiI5QljpjJceIYCb5P38OiWjO2Xv/+38e4deKGG7oiqWcLtea8PVL7Ptu5n3bDL+QpNPV8+nqjuAqwlnkfL4Qp2mYmCNi9M55iOzhiojaJyLatDn28Gk+JkxEI8bwQAK1tve7CDKSLATMpja8hwUGuYVHMCOBjX0LVB1NZ4twUz6fl/NpfcSwSX10JI2w83lhWb99/fzrL7+8f7lkDlNFBocpkkpZMjFhzN7icAYRITESm+scLYIOhLOqCifT0ceYbR+zI5owqrrqdLNwc1eEaK3tj3uEC5O7B3hiLiRqBgi15Dj53vpr21iwluV8PQNQf0wpabksdVkO0B+L5JwOKbkBjhnEwHXFEHeH4xNwXEgzO94SRDTGOLaT3X30nsqamIhSQELNPuTy9FKXpfUBIOfz9XRacy0sFQTJOWdZakkpiRQP4pQdOMJG24ELmSKRmxIhGnNMswZzd++BighCwkLCLhThs3+XpEUguKuaEjIYhNnEUJ+6PxzdbJiOve0AVJfl6fn5fDr3LeaYvXcPL6d8Pa+XU/n86dftcYOSUuLRd/Oe+GjrMkVA0H7fPEGpqayVRCLcdJayMBdT02EGMOfUOc39YDtCBLMwi+p0U1fyUJZ0ulwy89Qxey+lLimzWQbIeSLe+uhqajHq6Xy5nCjztg1FP53OeVnzUomJmERSKXkfffRBzCVl4Dw8IWXORMRCRKUUROy9zzlzzkeD/pg25JyxlNnmVBNyDCdEkXRaks4a3755UKlpPdV6Psv6VJKgW2Y6nj9VRTpCIa4WgXpC+45/9QkW7gQ2OCaRTh9z7FMnyZJqSaEGOnXMsc05jk09BUpSSqnobL310V076uQlZ0aT1Kmrxd6atM6RMeR8PidNfTQb/e11nM/16Xq6xQDzOca+7eulmHk4UrAApCy2kIeqw9SgMHVDTu7ObMLsHKPPcD9mxpNpzqazmzMTARAgAREYpZQiuLWt7Q83JaHBTBYlJSmU50BB4Dg/rSDsyYe3TiqXfHq61FpzqSxCiCSCRIEayEBigX1YS5CyJMpEJMcfItr3/dhn2/fdzEopInKwc8InmCo6xgAPYZ46Wtu3bZtTISCJTFgWvhwhETfr0cFDRIiB05GRRAht+z3nTAChHhgSFNoRFGKAtRibjT5tyigAAXML3a09KIIYPWIMNWI+nS/Xj5zo3vroe+go4sJMgsuyWkDw4WRzMueSxFWFkMRjtLYlIQCYox8kmntrifm0nDJwmBFArqcx+tSmQQxIkpBo6jDzJOWoqueYwpxYuBBRgHsgEjNiQLCZhKvb1NmIKOdsio/H/X67C5ZwSwKCeLqepchjb482DBUSr+t6Ol9TLpJEJLMwECORhSNlTKAGcxtuxu9+llS5LAEoBxflDyYuHB/FQ7J5fCOn6tNS2GWMxxzTtENM9xFutWQz27fH21eeeBp4aZkFYC3fpzHMcvjDv792TEO7owegzkmBKuQxfLYxH3Nsbt1h3n+/W61IMeemvvexIwYzDdVt35KkZDNdX6RKjEQ+FZUxmKiWlEqZjoYcRIiSJDPT3uyoIwElXFtvCEFIAUZEb49bFkmUOQuQZElzNpEMTIGea6pL1dnMwA0xUITLtfQ2IowCAixjFeI46MSIEaY652zBQAyAgeSqo7U21YWRwZlzXithTjNNsId2LJRqwlTyuZAJIAcxSZaUAGEMTYkQeZoiolR6BAUkKQtzEgQ4qPKIwIhzjKOfoqqqExFLTgiO4qAQRPX8lBPeb69AUjCpQd/3vt/OMQr7mnNO6djuicBuTkDaumt4xLJUyRdmNtUI1zCflhjH3B+3t6l7FmTGvX8dk4lCdSBZ2zcAQ4Ixx7btJeVLTdYb5XVZTikvcwxAZObEDIBj6t46BhJAt0GKyMRIrT3GuDOj+7S2uVqYTR2ZBA3aY/c2Sk6YqNYSABYxbU4z2xWdhViYdLq7AUZKTCgQccCphfMM6zpM9RhLhdtpqfVy2bfHtzcf2EouzOAgJcl6WUtKvT3apjoxy7KUxZDmGNabpJVZkpAII6EFGSEm5krowZJKycOuzqU7M6K4TiYiPNKE3FsDRDNjQsmZmN3jvn/JiYZqd+N0UuF0EtbW++9Ljczko/XbF0Sx08VJQOq7Dx/X81O3zk7Wd5i+lCQA3SyYFRwYBAnDAkwSAbpOzVKEEy5q0QNw6w8G6taTMAYQQk5MFAAjAoQXKRSc1GmawXHi6E1ErosEOCDsbG17MIJwuPvsgwsvOd+2V/ORsnjgijWXzO5qk2Ka7kHsxJIXTEV77xrZxZlFSMTavj32N0JIKQkxIgplRAoaiv5dLsM5MZJ7v+29tfEYo2k4JcnIWRBdcUK4CkQC44ySIZsFeSyBKSFzZEHJ4iTq5ACYCqQawRPJWHJ9CaDphJJlzCkiOWcA2Pd9zElEqgoAOeeIaK2F+SEtS6Uudem9HUyAw+gOMbfHff/6LU/5yLmsEuH7th1mszkGuhYRFrYIH0OYIMLdgZGZwyKXWpal9+YBLGlZn4SZCIcCRJwvhRAjnAmZCBFqLUAc4dMwwuFwVxKbTlWlLKkUCENKFyG3uW/3ZiNcWcTdESHnghAQDn8s2ROTIB/btPu+UcpIiUuttVbk5EIIh/9FhFMuYdZ7bx61LLVWCAhHZlnX9dCuWG8skksOCHWVJMig6o99CBGiB0BZUqnPQPH5y1cYdL48LSlJyZQrIDsn4AIkcOjKU3USm+4YOdG6LObobgAhR8dIVd39qDRqrUeZcVzFCF+WtRZhybkuB4psf9wSp/V8affXfR8A9P7jR6rX+/7gUj+8f051MY0jDplLqankxEjEIiI8uo4xDEI4juVqkbSsp5ITp7SszwAhKV2fkQmTpHAdYyBETnIUsgGkNk3VYhpgAEQgE54uFxv9dnsrwsu5CMrpdAqbj8dQVXADPBpSDgBEXAr70NF1eiBaTkmEU5Lp3toOakDCKTMyAB7SWJacAnWORIQACNhHjwAjPyDRTMjBBBE6j39SU59T22M3dwd0je4KESXlkrPkVNYqpaa15rJQypDPpS6U8rRo0xUo1TPlRQNcuzqg01mEAu2gMxx1xVH4i8j5fBaR76+FUo4VSLfNHbKkUioSl6oIYKN70Pn6cj1fdeyGtM+Zy0KEfQxOpS6nLAUCGAIDPAzcKUJVTfX4jWMqIWu4BdT1lEUQYr28b70Tc1mRAEQYwh12M9UAAGAgVbcw/P4poIDj9o5jawDh1LdH7yPXHBHT1F0Pm7Zrn3/UecScJWOCObqNHh4eak6lLHNvbW8zuiOlXCyfhFlECJmlHOpmkVNJojZ0dAjsMbt2mGYQgsAR4d57vz+2bX+oTkQSoUAMU4uI0GGuY7jFcjlJWblUzIVTsbRMkNF0Hyp5PV1fKC/AiQA96VAnlqmGgEfrTY7q4uidHmuPx9NZSjmOqczsevg7YEw1n2rmwM2iT1+XU018e3UEqCUzp6naWnt5+fByfY4AN+9tn3MIYUrZ3FpTU2VCAurdtjZL4gBWU0D7HkklcMSth6muqzCyYQ5kDQ934DwNKEyOfXyScE91IQCzudS81LIzq0Xvvfd+JD/iaI65GhBEmHv0gYGFhQUFM2IckbLDAMXCh9TdI/becs4kklIOB9OZCjEiIBAnyRRhycEhzIebTvWpw0afYyLR6XwhJgsNCAC3OQAWBFed+7632WS5ELOTGCUD3jrMuU+PlJecTyFVQTyIpCwlZcCpZsOO8ND3SvH4e4zRez9KjjlnrfXvww2RxExI3PscU49BQwCdrs9B8fX11vbRbY64//jxp+d3L3W5jtE/ffqESPvezCYBLEu5HF1/dYBg5nCbqmFayhnc9z5oYpLUdZg6C/cJpl6DIyIwSyEMUJ0skktiFpYELBYY+P2GE+beR2+bsOTE+z5rrSmRjrZv99YmR0qC4A7gprP1ZgBMWBInEUKMiMf94UDImYkDUSQd7UhAUgMADBBJyXTO0QncwaYOB2cRRgqdMaaZOqAkuT6dy1IC4r49Htut9w3Ckdzdp81hY2JEGAISC1BS5wlZScpSr88vkpcR4QZmkRBLrULsoKC7uY4xWmuiqsccY4wx51yWxf/4c8wd55yMLmkJhz5GAHOuzLm1Bj4dwzGh1Jrz9bKUZf367XX/66e1ni6Xl97nVL0+XThngwhE8DigLcSMhESytd38yKGmAO9qAK6qpH6+PiFGScnmjPDEEmCBQJKQBRCmedgM4gBkSWra984INs1UUy5E9J0nQ1xKgRjdD63WJkKIuG0PZ8hJEmfVQ0vpx6ZDAALAsqzr6cSSEMkM51DXIKDZOmII8XRT1QhvvZFQQtI+QxX9gKH62+2O97uB9zlMByMEoLtNVQ0HEQp0ZOfUjTwi5Ur5dM6llMp1QcmV2RyHhZOoIyMBJZZZUmpErTU5XpsHuDjnzH/8OYC4xw+FsdQKQEPDgoSz1ISS++NuNlBKIpnRfv/y+fb6r2F4Pb8UqY/7nYh//vnnXNLWNiB0CrXZe0PEUjIxM5M79KEiLLkAHNRlRERhzqUwQhKeEKYKCBgIeORn3QMsTOOwRrD6+H5HH7JYCFW16fHdykPhnESg5H0fB9qbmHNOPvc+po0mLDknpkSMY06HSDUz0ZzqxClLTkUEdJorEhdCd5869XhVp1oSy+y9zeFjoJqpCkbKWRJPm3vvqm6uY+5lkdP1rKpvjy18tmnt0bEsuSaRZTk/n87nZT05oJqb47AxHQjAPTx8TM1qeDTZI+R4Co/H7hgx9t4PNNFBXck547FzEK7mKAe8NSXHmWYAckEfo+/32/bocxYuzNx77/usdVFVYsw5l7Ui0mEmzzkvy0IE27YN1bf7VmvJJQGgBvrovXfPKRfG72/10XsnRADXqUAEKSOAIyIe1sJERGYIru4WiASMSJLYbLqZme777jqSSJI0EbdtE+GUUlmw7U17MxuIlQoLicixWehzTkGmiD7mNqcpCUlONeckCQhNp+yd52x9jGlq5gDAJMBgClM9cKiTQyARMvvU0dXDj9vaAR3ZkFmWXM/l9C6frpIXyQWQVW1M/w41piAiomMzJ3rvznQUF3IMLv5+rjnmG8x8nF2PLAcLQ4CqewAHDT1kokCcDuHD1rVPZeG6sDb7/OXLnR9Pl5da10+fPq2X9fndSyC20XkYERESQKhaRAgnM5tqQHRMxCLm7A9TqpmoFDfobW/bBhCI4OaIkfBCnAIQiDSAzVnSUsocXXsgAsbR12QAR7Dw48bTJEF4Cus6OwAIy+vr70yUkkDAnHPO28vz+1rrMGjdfE5OJY6agtAjtjZb88S81LScUioLSqiJ3e42JmZAWMls9gbuhr315s1YCIUqL5yWXHLXbQwNikAaY6DU9fx8ev5ByouUFUnG8KndIywA8CDjhluY6dGr5u9vMgcAQTwsGaGqxzE153zobP/YJfYkFRHJdV1PuZ4cqM/h7h7HJ5mAxDyCEDDmnOS+np+enp5ExNSESc3a/UYRJ2SE1HuHWxChu5dahk5z19aORz8TqPYY0WshDISIUI8Z5iJ0RMmOtDGxIKcg8YDW2pychFKSLPnoZrq5mYcHEddaXUNnA4B1XRHDTUsWOK/btm3bLizXy/V0ukJgABAgM6IkIt62LeVaZMmS3Tgcc0rlsML6VFM1QzyWBVlKWUTCTn3b2n4npmk94LhDUURSuvCkx/42dAxAzimV8/n6cnn+iHIBSqrzCEanXJnIAyJQbZobjEGUAMHchJCZc87CLCwsiBGg1hlxqv6xuRgiwsKtNVVzoPVcllqnxZwa6ACgHkJEKCyFPLu5JCMjQPjy5cvr69sPP/707of3pmYY6/lMwwFRp3p4zsJ0RBoVIsacbg4nSClUdfT+yIIEB7eQmYJQWACDiPoYw6DUmkgg4hDf55RrFtcJoY9tszGSyJzdbGC4h4F5n3OOljlKLqoE4Ul4KaV7HKNWM0NMU6dHkrxIqUi8lApIGo5uIjnL93bStGk6D/nl+XwFh/7Y5mgRSMS5ZJHLsi5qs8/+2DcbZki1pFMRYLTHW+sDgv/IZB2YdbAIdRMikYRMfcyUkkHE0KOrQMFcShJxN0QULHVrrZQSOeuYpS5zTkc8n9cIaK3lJMEMbCmlWpPPNrbmYxIIO27bvDcNCErvtKdpG9fFx/6f/9t/vb++/eOf/vHDjx8v5/f3bURQ6NUQLRRwCoPq6O21b/dwLZyFYgyFbRunBCzbuNv9EWUR4b53yVJKGUSIwEmiRQIGSuERNjCAEQl6uIE7AGSsu+17+wZEjqRHuCPIuU7y3z590v54Pq3XtdDmC5dpwwx8kV1jhlI6cVqBq1MWzjnngAh3CA9uxjrVIoCRUz0LXmyamwKGlOJ+66NlgZBQc0wSkgATYfGk9zm3YVVSLu/Plue4TbUclYLAzXm0GUgLSwZENRAkcBytn5YFdZtzJmYPD7c5NMLDv8e9aI7JTEutZpZSIsQIOEpGndp1pJQk5T7mGLs7tDnm2HW6e0jNo48xjUhyKXPf9j4B+Pr07s//+O8vl6dPnz614dfnDzoHcMo5bds25sgJhoUFplRZcowJHCRZLVhKrmsAtjYIA4jIQdVFkJgpKC85YWIh9HAPQtQ59259oJsR4KGmJGL1cAyRVGtGjMfjEaolFwoNwKmG5vu2TVWSpGo5Mzr/Yf4R4oTERwEtJc2hboZMzDinjanucewqWcS23Wdvglhy6u0xx54TE7P32fsIiCNo+Ha7DeHzWvrUMeeyXEDSVNtb8wndUVhSSkc3++g4HgyCv6faEDGV42XrbW8CHkVS790tUpLeuxx4Oski+cim9jkdMHDMqb1PJN721vsMwCMQ1vV44wqATQ3E9ONPf76czyT5v//L//CgH3/60+m0qDoIT7X7fZMUpa6A4sCST8TiCiREqW7jQVyWBc0UUZDIXMcwN1RxEVcLSkyJ0fEImoC7mWEAEoFHSZmFGdP3us3dHGgCC5/rqSBmxPbIWej5tJh4+/3T4djo09kh5+ooJElSIs7m2FqP4FoZ4dhMkqXUOWfbNjUb3SYahUW4u3HhKokp7jYiIqWEwoC0jxERl8vl6XLRsW/3W5+a8pLq6nmVnFkKppqBCOQoE8YYR1vm7/4UVT1+yDUrgHmAJNHRcs5JeI7hEDmJmZ1Op5zznGNORWROpencxxTJlItHOCHXCsBDp4YrUaqnUy3knqXut7dEOBX+7a+/nNbrP/3Tf7hcLkd6wzHu99vex/N64XwiNWTntESgo4FAsCAWYpJghx4hHpDSgghEyMQIYDM2a+aeU46IcQCgxzySn0KMQlIyOZGKQui0YTr6tF3BTciZGYIiKNcF4XltYx/dwAPIgmouQYW4IB4Idl7qxczarutSc87hjkDCKaWKqHPq6IPRAALA970rBmOUUuboj/sdWHIugXx/PMbYz+uScuE0kIfa7K0zLRDIgMSSMIXhcfGOpcTvG1ERf1QNEREGqEPNbamLuFnbN2YmRFMtpXQbh/Bh37qZkZBIblNb7xloWQoCclngu2mHATECxpi6zywieZVqbd/a/R4OPz+9lLo8HhsTLet5N2ut5WVJZZkaDknK2SnPoeaJmDQwlZO5W1d3CeawCD4IZSicAlzt+6ygpwl/KA1IOLEc/yNBaGEImEpNiF4w6xxjztG3/aFqQpLLAqFjBgXn9dr9bj5FEpB09VQIKZkFBKbMOS9zToBgTuE4hz2sAQQBllJrRTdr+9vo4yi42+wEXotExGPbLCDlGogppZRp7N3dAsmJhynkPC36fd/8ls8sGdjhu3pI5Hj+1nUF+P6NOwLfe5+qCojRhzDG7XZflqWUMoZDAAGa+ehzTnUA1HAGi+hTh+0GnHIJkt7aY+8AmEttNohzyXV/bDY1IAGXvOJpWdrQ//Jf/+sPHz78+PGH3vfNIyCer08sfN92ACjl5IB9qgVlyqpWcnFrFkxCeamMgBBEkJPUWgJ8tDYYAb8T0Y4YbTo0rwDuwUR/hD+JRJilcqoFwjQL396+5cwpSWv3rnqSul7zNm104LQYUN/7iRekOEQkhXPblfioKWmqucfr/c1dE0mppZScJUsSM8pZGBdQ0tl77xBWS9WAPmzvgzDWRRzg9Xbftj0t64effjCS2/S7Ym9D8zyl5el8Pi7e8UVsrf09VXpcRUJ0j5S/x94EzIpwe9zQ4/x0DcCI2B97ACALEhPRMGNKLMksAFgjAolLSQ7DvLkBc05ryetUH6rIwmU9na4U9i//+t+va316Ot/uX5HZUjmtp3BVBeasc84Z7nrfWi2lrOfeumS5PbbbvX34+G5d1ghLjG4zJa51QYh5+EEodExmXtd1qVWIwmP0zkSSEkSYQp8j5kBgBCRH1eGul9P59e3L1LasOZU0Vfe2GwpQNZSUCwFPw1Cv9cRSp8Ycg5ndDrCag9u+bSkLCu77Y2/b5XwuWXRCmyMJrsslrLy9fXvcOpGgCBMSmen49OnT6/3eh5JkPiVelmV9XtPyHPLWLCiV5YSIx8fv7e3ter3mnMcYx9fxGBpen55ia1trNiciCrgnJkyJMEznmDanSS4A2PYWAaVkJ0agkhZlD8C+DZJEklOhUJuq09S9m6HF0bxFMBuj3b5+vm/3sHb6kgTn+Xr1GY/bJHwaw1qbT08vCPLly7cffvyx1vz58+enp+u+b1PNEKb63ju4qVAimFO/ffsSbhHx2G7MJCJzzn3fhMgAS8ovzy86577vSRJLuu+P3trcBwJdT+ckPPucoWYTEczm3rSkk6MBV8wCRA4MKIASmIIEiMHp2AYkOpw0FgZPz1cmQog+VHU+7m+wJhHORfq+9fuN8buJb4yBDpLKtVTTpvOhU9/e7s5Cy/WJ6/L0XpbzQ3HeuiGnWvv99n2PWg6J3/+x1HZ8HccY7kYAyIyE8nh7FRGWxACzj9FVI0gScyJiNR9q0zUAAgiB/ejDEZn59mjDDJlrXRmZUQJgqM5p2vfZt65mYW3M3jfTalN2a0yMCKOZA4XH69dXYXFzVU853+9ba3sq5aV8OC1La1vb7m7jVPOy5LA55yBE1ZnSUlJW1VDXNsz9obf761vJmZn3xyPQy5LDfe7dtJuWpabzeZ1zdyhj7lOng6uVqRGURDIzA5GaMyXihCAYjHRwEo9xx0GNCus6fCI4IjBDuH75+oqgjDFm994Y0V3NfE616QuQkJSy/PjjD13ttvdH9xkYkoPTDOrmM0LVxu3+DFFKOc41InKMe/9+TB1jvL29Sc6J4Eg1C7iFo3BBRJvTzVnEzUXofF7MYW87qLmH+VQ/VABJ0bvN1vp0l5yJxCHcx7Rp4SxUZJGE4AOFADSlQLZt+2aQ6/np9cvvEXK+vNzf3h6P9vzy/nF/jDFSltvbjRJTymFj2Nza1vaHEGzbVG01S04HaCsnFj649wD745FS0jF/+ctfw/1yvlxOFy7gALnmxFdrU5DIjYVSqoFz6MPRUi6jkzoCComQcAQQBlNmToHkgITkNhEAEBgPJwLXJZuB6fQwszl6n2NHtEQkLJFT2zYmuFyuc2hTSykDUG/drCVJz88vWfHp6V1ezve9a8duKLmC4WN7mEBiPk43Y4wjTJNSOh7K0+mUUgIMVe19mqo8Xc8RmEoBoKmDmZHZ3OeckgoRIfBa8lTde/c52pyYgzRGBBEL8YF+zykvtQICUAARIAXCodOBaRZT5x7u63kpiXb4bk7a2j1Jfdzvp+vqZm/fHjnJ+nwZvb/e7pnIdEqSRND7o/WhJZ9OizDpGAxogOAOiGDx+5dP7nZeTr3t337//Pr5K1fIZ3q5vlzqqdaMDmqjtYHsQ5vaQIplrSznfUdVRSIPMI+USpLMksPB3QEYwwHjOOcDBUGEzwBjAfQAgFwognSqqi6llHzKQvtjm3O6g3sAECKNOXrbEOl0vnBwkNy23QWoZod0jMbGnADzGNHnnI8PobsfPzkupEegTQrPjOggjPQ9BoFIhCllDTL10fveJ0sWESTEgCzJDboqAUaEzTn9oMhJSsXDet8tjjGw6Rxt20xnzhJOqkN15JSWmiPs+ekantyEkwTSNGt7ZxoelhJv29b7PsZIpVhYb+02d/SRhfqM9vnR27bf2/P16en6ZFMBYSn1WFxy1nVZay7bY7u322YjLOAal7IKILi56b7fDTTiO/M75TJm94hj5Zs8WJIcY2FHd0SEtj/M1N3cFMAhDEKZKScKMHcN81ySCM99b+3hRELQ2v6474SigSnVdTn1xmbKnE7LuXCJtAJxWk7ndx8eXbeuEIFEOvT4pv79o3gEoI4ag4jMfbQtJS6HYczTShC5rlNn27v45Fxvb68plyTZ+9Z7zPW0z9F7V4umgynCeEwdaufLlVge+0YEkIQA2LRQQmIx37e9v95I74/5OFtbf/gxgEtekmQNHF3baEehPnQfU1nEcfQ3QcL+sNuXz0zBFPe3++WyBtL9sf/Lv/y3peYzjE9vf5vvf7y8+0HqBdbr+z+9MMJ+f2Ofp0TXM9/vveuG+mrdPD+7ZCPX8OnkniWvIhyR3Eh4jXAMoUMxYsNIEEN9BLQAktMCc9jYuvXR7hgqBBVkBs3ezbzkEhaCnNI6x2itq0/jqqQIuJ5PX9++fXn9Us6Xm7wEYM1P5pTS+Xp57yRJ9cxxPomDP8LcvwfYHo9H7/35+XmM8Xg8RKTWepyzWhRwEVpSSQJIrbVpj2MtllJalppSEhaAUJ0WPl5nM3MP9RhjsjtIMtXZdZaeCzIcyhUU5D3G25cvodrv3758+uvKfqp1ScAEDGDqZjM8/ACHIVm4u6uNPoa32Hd2WIX5119+62P78O75fvvGFB/yorq/3W4//PDj89NT0be+vZVaz9fr+fmHvF5zqZn5tNR2++pjI8DTuhaLtrfZrUnLFSXV9VxJtOtwQwCOOPKVGTEAGIEQWW1YKKIHGKCGweV07eBHu40Ixr6jMGaBiNnHGFMAWyg4MJOaDh3hAUBlWd3dIDDx/mjb7Q3XFybJdY2QVE8plUfvNYIhpqqDMbi6H6ngY7L0eDyOGf5xCccYiHh8tjz+CAuzMDEJiXno1H1vc6p5HLUjAPTRA5AYMQzCA4IiEPAgv0QYEZ7WRW1uj1uoj7a9ffmcKf7pz38uqGP71h5f326jLOtpacJAx0ZeypzEAPc+YXqATZ19xHD97dffWttqTn/9pTHZDx/f/frrr8ywb/1yWs7np5dy0fFsmJb1UpdFciailHMqAtr66EEEzuG5lJpLLbUSJVNSDQsBJwAQzpKymhNwIJiFqrp1C7cxEJ0TEoObffnye815XfKpPtlsv9y/TtPBhBGqEw7qg81wEGD/PkmM++MBiNfLdeqcrpQFKE33lEUkBWViUndEJmYiMugAkEsRKMz8eDyOpeA5p6oeR9ajG4CIjEGgrmCuoqYeEBYeZsdSEmIc+HrAJEJE8/gOqLsZEzCimUJAFk7M4T563zhGe7THnpgZ7bKWl8vpTx9fQre//dvbbXRL2dTm3K2yELEgCxBiOJiPMdo87uo5v273//G3/0EIWaTvj59//OHLl2+//vUv754uOXHmDEFAVM9PkleQohEUwMiBCIClVL5erMl2U7dRy1rLKaUagW4xLdxBFdVBDUXNUREZ4Dv6Vd0cQt0iNJEwhLrRHI4xZgvtS5Wffvxxv99m7zYnAaSU1NR0EtERYFfA3uY++rbv0/10vrRpNme9niSVup7ysnpwgHiAqt/uW64Z+YgUQErHRvb3GM2RDT5Kz//T1rCHgbopgCAnIG+9A2CudT1d3u73AHBAVQ04KpJjuXeaqUi2iN47ICMnYjS13hvEzAmvl/Xt85f767cfXp5+fP9UEqW83pa6lYIIU2fyYTbMmX0GCKBEOMBkRkm4N70/3n75sqe1vn37+suvX7Nwb/vb189PlzMjfHh5+fD+BwK5t/3p6ZTqyYEP8CDi4e8zkVQv58HEBE/n81TzQLeQUkloe7tNdQtmSY60T9vnJsKJM6NwypxKbxuR/CEKBkDw1tponJDQ234XwtOpDoo93NXd3caUjJyJEomkYJpuT+/f6Zcvv337+k/Xp+u791+/fNmG5ZpAsiM7MCAGkJSkbqQmlIhZmI/D8BFmO96rR3/j+7sUgIgOOs/39JRHSM6BOKZGhAMGYMol5YyqzDLntD5cAwNKqanUvQ33IEERSZIIvS61JibU+7fX++3tVPPz5VwTPZ2XsCmJU07HpA0x9GAogLEwSow5ATwXdkhj7J8//9a0krAsVVq9vb1++fxZIH7+8ccff/zpf/nnf15KFUplFZQ8LQwssbd9M50MC4eaTT8AlwiMpmShPufB1UzAMVo34CysanvbZ3RSItSDVI1ExCIMpqHaIyIlDvTH7fb0fLlcTvebRjhG1FoB8P76mHMupQZZG3ub/WiYBKEHlPVUg9+2nkrmelJHDTouIUlmSsBCgWATEFtvnOi8nrs+UkpPT09E9PXr12MqeVzL1pqqLutKaAZHqB9E3UsprU+LOMAkDmAQ2oe7j4NeBgcINMJBJOeM2WI9nfc2t9Y8YN8HgYC3r1+/Msaf/uGnj0/XTBFuczRmKcuahKUsAKamFF6gqo6Sk5maq5mbDdUx54h8CSYQnwAGcHl+elrWf/6f/vl/+g//8ePHD7fX15zkdCoGpu6BNHWAofb9nNnDxvYAU3Aj37a334CZKQXyGI9AlnIqa3KkMft9e9weD0iSJOfEc04KrbUSAAAjERKBm6qdSnEdo/ddIAB6a8xoqgRU1/V+fww14phurW/MudbVgO5bH9OB86NroUxckCiXtZST5DKGI1MtSx9DqKiNMWYMM/Xrejm+f0fhDwCttWVZ4vsmqJtqziLCRzdHhhmotTmOswwGpFIo5db7mBMMcymcJOWMMt5u9/u2a8B0mAaGSJwJ0aI9WucYOeen9fp0vbrpnEqOc4y6nJ4Rh87pwe6IIJJyyR6h+kfoCI+TFDKzIr08v/c5ze1yPheEf/r55z/9+c8i/O3btypiplOZMzMxIkUEkPW93V/hVLOO1rfHdr+D39C/uRNJ5lyBkqF0n0AZpAZBsIGAE+46AzlJ4pQkZ0IM1zAM+C6NPeIE5tZai7DHtgPG7LPkUssaJFu3pSALkyUHbOYeYMAgCYEYJa+XZT0DS1rO6/nKnP3R3FEdHSiXnCIBkdo4ujOPx+Ognvz9QTw+h0fa1N0tgIUO0pogsTsAkjA7gCGoWspp4VMADp2AGCic14QSW9+HkaQgaOqcitTVPdKiBIM9LM+6rqUUGOE6tn0ChJRlyRlGb60zp4N3zJzmtN66mkPgwT0VEZGUlsv55QOZ1VzXxGTj3fn0+vaawTliYDxfr+bCIHTs7pqZDrRx/7aVl6u47qPNtj3uvy9leCBnXVJKeSFMjzaHKbICpaE23Ujy1OnQK2AEIdtSUsAhFwARWWqlrZEk16OR6lLqtm3TnZGCEqdVwPf+qt4ASMPnaIEJuUhZARNgKqen8/UJRDTAgJCYU57T+5zufsqZBQ0MJ9SSj2t2FPgHHfFo3xDRcWo1MwNFYGYmdDkKmlRyymWqmoeDjanEIqUCi5ljKl1ha+rIlBPnvFICJJKkHtveuaxVqu4+9W2qmUdJyUP7NkgkCAIkLTnV85oNgMIPZmcEQko5HcdLyufz/vTU9vou1Yv1RnlKYhjwr//6b5dM15rvX38HHZn/XbkUTwJExBRgELoI7r31O4B76Fhrmk0iZk7Lcrqsl+dUzoqCbLc2LThIWJgFhxuXIpT6nGM4EoErhuscczZhJCScqh7q4KbuGg7qoMFtRuBUdbWY08wmp+xBvRuwMJIrpFLqclnOL84lPJChDzVn80NNTISYUiYOZjKj4/15PHnbth3ndgA4wt/HcMPMus4UKIIRIYCgpoAkOQ1Vc1+WpQ3d9v2QDkpCzOu99W2ok7Dk6XFaFgPkVOYYw9q6LMSGUzgv58vT6Xyh2QKOdzR0iwBIpeTE2fcxp6q6QQSKyPXpvSOaB9K+rufLpbfzj3U5Uzk9PL59+XT//Zf759/+b//X//Tl6+//8r/9f//dTz99+fobLelJMFxrYglPGB4j+v7t8RZupjPnsuQFSep6Xk9POZ81pO32emuP7m02dbSg6dZxXxcGsG3bwIEJy/XKQgAx+r6PrqbVB0QYwFQ3N4AATm746HPrjiDglHJGNgBGkkAA5KmAAevTZb28K+vFHM0mUqibqR33Rc75fD6xJAgj4pQSM40+jgbN0XJblqX3fqyZttYiorcGtTAxSBIiMbUxJyDmlKeoBwRQKSWl3A78DZOO2aYFJxF0h6k6A/a9na/1cn1BLmZhvhEJEudcaykHANLD7ts2zKGUADZ1DjPzCBQ+8Bv5dDpN821vOqeOaVMt6PVtq5mBpPUZQD//6R++vX37y3/5z5U8fvqAYHvb9HdYsixZ1sQl4ePtdd/v2hsTMaJ1f3l5R3xoMFafvPe5T8jpqmF73/e9A0KIKEbrjTk5ABMBYO8jrTWnNFIyGzp1mCYhJlFU84Ntzh762LpplLLmvLDFaM2Icz6RpJQqyHo6v/v5T/8IXPr0QA6M1m61rEkKwJihxEkkjTHNupmzsDCXlP/OHvp7vPsY9H8/QNTqKQOyO6qZKIYTCJG6ElPG1PtMknJKmeTAoSgTAZxKApY2bElZ1bqOPHtd8lpwqo1N+7QJ+Pvt8fS4Pi2LO8wAAKiJEoOOe+9tJGZJktN3UB0X56rhm89fvu3/9vvb4zF+3f+39XTpyD7msiwfn5e5ff7//L/+nzzbT3/6c831sjwDztv9byqJlhMgOov3DtNzqsvpbBBAwOs1p0pM22y3x5c2zSWltIKTow91BMlcKgxycFU0W5d6Xuvvv/6y13y9nAB8zoEBMyhTatt99KZjNx215t737e3rmLMvNacSHF07c12YEbPkp6fnjz/+/OcPH3++79v+7Rti5MIprUtdhFMWHJ3cbH/ckggzZ8pt35/eP9WUjqfwmGMc2xY555zzsZe4rutddeiYOgNC3C0lWZeVSTqM0XqWFO7t/jgWRxKJhXqgSHJCQGBOc/ScE1FsjzfXyQIsyGlZz7jr/NuXN305Z3BK5ZKlYKCN4V4FLS3IFEFbm6lkovzt0b6+PX7/9va3b4+3CZBPS46na9IWFlxTymL/+t//+mi3n57OQ9tST2t5HvKa0dmnoMV0VzulWlAwp3I9K9MM60Fg2Ef//Prp9nilkqtczEBDkDwlhpDCOZ/P99fb4+1GCLTURDHafbYQMiGavUME5Boe3+777LvPffZtjDRn3/s2RlfvVkpPRTmtaWGqOZ/r9cNy/SDlfNBSCEN1D8ck7DrVDcLAh6uXUgiscGYus41Eufd2LGAcHVBmvl6vx/7aEYNblmWObu4Hi1RSSr31++2OgKqGf5zNgtAcIqCsS2saDkAcIPC9aUu5Vk6p7Y/eWs5ccs2LSEo22uvjDqE/vlxP63lJlMJm2wiJswdKBKq76feB1Xb7/NvXb9u0XOrzC/c5rx+fTDExlaes20Pnre1jWU9j6tvtniQtp1WKi7MYJE6EVjiXVArABFebKHn2dtteM6Sjh5nLmXJxT9s2tn3rLQhLLqXWWBLerIX3UjP6uH37ve9v7k4wGHHfdwAoNRnhuH+ZOsfoc3Y/tk9lCcqWMqynxMTEuZ7rcl6X6+XylCRtj83Nx2hgJsgeOoZDAmJyc1UFpFKrqZorM+Wc9n1Tbcem+3EtmXlZljnnccz5HuMIEGIAGH1I31oEpJSOw8ixGjiHiqT1ejooZ5lKpgwiwwIDfSggHCiroFRWTkyOONVEZKlPYydF7MHJERVPpaZLxr72voMhSwoPU9ss/NG2ocH5elpR0tvtYfc7QDpOQ7UszZ0zvLz/0O6f5tDz8zOXkkuVAmTIHgk5CSwpm87D37Rvm8+9zXG/3dA819Pp8kJ52Ya93vYv3+69T0Q6rygkpv23v/06Wq8UmWg8vj5GD910zvu8M5PqhIACMj3m/haAbqoeUs9djeR0XlfJVXJJIgiYJF/OL+fTlam4BRP0bY+wpZSc0rDZTRMLEk6fABzuc86cEiH13gGitV2Eaq0H4ORQE7fWvjfbRI5BY0LiI+foIdY1pZylBKDqjhHLevm9f9n6fs2JmA1QysJ51QDfG6AYuCEIZ6fgXJeaEaLPEaAWHpJkuc6+fX20Plkw3l3Pz5cTy4JUfMz9ABb0ocCcsnNyYsUUTiBluciOXpeCkdQQkUXS+w8//Ot/+X/b1PcfPgZg11mXzD4IQqQsiQmwtd0hcs0k4GDnc1kKja2DFBJxF520b+1201ryaU1J4nH/pW13sv04Phg0TrIkzmseM9yMKYLZzHJsXUeCLuVEuQgmw/RwoLKeX36QcpJczlkkgEmul6fny5P2OfpILLVkwmCE2ffRmzE5BiMxSUoR7kfQFCGaa61L750IjwfxGPEfAoWD6fZ/hBktMIIBM4s8X59bG/fbpqYocrpckTgQiZORTHcSREokKdQMiSRLkKmlsuTMbd8NmSCIy3k565xzNCbBvDza3nQkjKa3r/dGCGGu6n30rc1pLiWR1LyetjaGITIGJyRZqtSytvtw9VxWGIbMSZJPXtaFRFQdqSAkAEMWSnm0TQHczfpGiQQg57RcT3q1PqhrHh37oIC8Lk8iATFmf2j7BrqdUlJVJl5LWk85pewz9U5zDubjyNrj8dVbQ3VZ14SZ0mlyETktzx/o/DyD0vkpJ1kCAFAV+vCcSnIQIiYKHVPnaM1skqwARCQARmRItCyVKMw8pfT+/bvX129m3+eLRxLuKDCOVtzfd4TRUcf35Q0BlLqIQIwxNWKabdvmQMvpVJa1H3FHyQ7IkhGVEmbJ1gYxs2TJFu4WgcIp5ZTSw8zdSi3mwYiEce/9dbuHGXic1lXy6bxcgQWQgljqIiCpLuqxv7454poXBIagnLLANN9NbU49rFilZBJkZEI+hnNI3IZOd4gwVUaKMBzIhO7hQYBswW/329dvWykpvKt3gbYUSEupHgCZmU/ndanVPYa71GQCgBARoNCs9+0+nXAOWE6GQuX07vLh9PJxgjy6UlqJOVSPOq9PNbWEaGa3t9eY41BUMQlJqqUezhcmPvaZ5xz7tiPC3nckRMe/d1CPB/HvTcq/w4kuy/m4qKoqQy3XSmGUQoCGKSCXmjgVZE54wKjT1mcqKzElSoiElAJgqkrKRC7IbfTee5aESEOHcEpS11oBHKmHa7jb1HI6SUrTHJEcsavd3+7T4lpPFjbMz+cTE8+hwrLk0u57mN1vty+ff//xaQ3TkrimdAw4hf/gFTIGEgsRch+7uQJz+LAAoAycOWWSrnGP3q4nOtUlk6M7RV8Dc1ncXDASgYUbeCkp8sFgN5VhpUgZ7HzfR6ks5XR6+WF99yOW09btRJFEhCS65VKen67t8fa435xwurrZMTw7NmNqPa1L3baHRyQRIhyj22G80Olul8vl909fjkPNkb5x9+v1emRij/ni0cdJKZVSAEDmMfZGAcYACI/T+eSELAmZIzQgEJGJ5xiEJFkoyN3nVAtwjMTES45NCdDmtDEFOablVG63fc5eUjqwGsvpCsKYCsIcUwPBA7s6Me/7bgAp5amWjDMLooAHBbTWfv/1V+09NM2+MYFw6Bw5y1IrYYw5KOeMYDZdhxm6gw0jyFLWVK9N09absz+/vybBNXmlLflgKBJUAkotx+47mjPA9XxGloMZNchEZjMekZxLSpeXDz+fPv65Xj+6LH1aglhLup5PZy5Y1pSEIHQOxjBXj1jXhQDCQojSsnBKvY1972P0ARChKVEiTkmSUM4M4EcsAwBOp9NBPTmeub/3V83s2D89fiKRMuQEAKCEEegQSMcZkhxMLQBKpprpsTeAQI/v4UwIRMRwt5htt96XXEfvCbCU+rjfhIgRNWhdzqOPMQZTmjolI5A42FSfqjnnlLNFCPGyLKpG4BiBYdoHoWvfH6/fxnZb/uHl5emMNtvjJqecSi5Vwt2cEAmdwQ0kZSACqLWksgYtFqJmW9+3dheR0/lcaFbgBUpBKRwpPJds5lvrZp5SuVyvAdD6aK1bDETO5w/WKdXrhz/9hw9/+ufmad86MBDRc5LTqT6fUpV1NB39MXUwBuWDCYypCJOEIzgY0Nj7aLvOCYiEAMgYQcQsHGYRfjBIjkt1gE5FjvkB//+ZT6Yfr0N3l+7BHogIhymyxBgzLBAwIMgRiWbbKRV3YyTEEAIkVHMhDkfTMbrOtmcPV0UkcrOh27iHuR36ZD9yU8A5b3trrTlgSqmW4vT9lZ+ES63uHm0bY0bw6K0wvH379uXzbwT+8w8f/vTTD2sW8glOOvtMBySjuImKpJwhAjwYSYQR6zbkMfqIQCYpJAmCughUTific8JFOJMh0d576xOZz6dzKYt5mKMHI4pWeND64+nj6fkDlUs3akNZ6ul0OuUkCJkiz0dNC1BsYw+wWiU8AEGEmUUou9PoE5FySmGWRIhJCAC8bbcAd4c4pBQQy3L9O3joGA5v23aoFg7qwtRZykJI7mbh4gQDAgIwLIgYKR2dWTzsxEBAe++nkhOTkCiAHIn2MRxBdc7Rw5TUHamkNPvcb/dzKUiccjGzdTnlVEqpB1ui997HqHU9QJBlXdoYSAgAbd8RoACgGwGBWYT97S//9vbpt3/685/+0z//+9NaLnVJkbcYEOph5nhwFVNOGJkC6WAj6QxICMV0zNCpw2ImFkmcMq9luTBdkqxJcrLe55weDkR8vlxLWaebAznMAErT77eHrGeuT90FQ+ppqXW9rOuCYfsN9kdK5LQEpJLY3JhAAYhxWSpxMkNtFsREMsfxi2iMMcNS4kB0UwAiwkQScUjLKaX099DGQQpGRBGppRwgJkBU865TlmUlEj0yHQAaWHJlDjx8ZG6OLinnUrvGNFftFgjEckhJhyJAyoXZBUmIBri7L+vlILMgQtu3OdUiHGOEBmE9rbUuSNR7zwAQIcyHPpWIPUIjBLVm6o+31y+/ic7/y//8P//jz38mg1ISGZJjSsIEptPdmRITB4R5BAsgGtJUe923Xa1DPFrrc1wuS63L01oviU6EJ+EimEQCMG7gjnVZluV6vj7tfRoUjf3R7tvYqZyH8X2bUlJCIgcfY4AFhLUb2ahp7e3RDxSj6z6HhxEfEj9ofW77QKDEWae5TWae2gGcOLMIQbAgEyGFUNKOvTVIIJymTlUNwAPSYQDDfbprG0wcEQgk0sFhkjt0TWUNAnN1cGAg5mBCgpKfutMEGH/ECLatAVHiFJKQikeEdUN9e71xkuvzU7eBMVvX0+mk5kHIRKnmbhiq5nabreZCSaZqKQUhZp825zS303kDjH77uKZPf/0Ljrcff/jwn/7871+W66WkhDBtRxELGH0wck3FLTBizrmPiSX5weWo0NyiJoK6/RaPu354t767/PTxuqZ5r9BOizCYheeSnDCYrx9/lPVp8mmQ0nrFaG9f//b73sr1+rQu67q6+xh7Rs9YxmOYSC7LtPzVpWTEmIFY65K9HFWdquqcGHY95Qifsx98GGIunBEBiVrbOVHF1IdG+PlUC6ddd4dA4D7aup7KumgYAIBIi1CWTJIlqSpziH7HmH1nFwEhAlh4HEgdxJKKh+97W3IONWAO4pveA1AKrUsFpNfXN7B5Oq8LYACquoWHURs9lwpEKaWSM4nobT9AR4coIsIBkVgwwGF6BBId9S9h7b1t27a3/tOPP7378HE5nQkikAOJjhrDfM6hw8LD3NUchCsxMweABnnANDM3lix5yWW9XN8dbH5eznnJYMNshAOVNS3GeaVcm4YGNfW3x96G5rrUy5WYEKmUnFI+1Agi+UiZIHgEhH/vXB8962MieBR5R20Oh2cM6djzRgQzPWoZAJ40PTxJIqJ99rIuEQBzpJT6HNApIIAwI4mwEPXeA4yzIICklI7WOBAS04GbDg9OB/sSmZmBrXcUGXtTVTwM3kQ6BkSkXF6uF+GnWuV+e/Qx2pjmIUlUvY85zbLkoYZMgHjg8kS41AoRqgoIFn7YmUVYVcOdkQMZmR3w+d2HejoF0DTt5gH/e1NvtiNHsmVZnkkGVRucDMbNi+zuBKqA+v9/ahQ6GzcjgqS7manKcIZ6EA9Wvbo9OAyiKnaGvddGJEFgd1fTsIXW4iwpmAIwzB3j0dvPp6/uVsqFBvQeGhJCnIMSmqQIHk4KjuWeLwLlaryNaX89Hj8+ns9Xw7L//tu/8ZZbbwuMS8SIiEgp5TmHu6+UCET2VcRxmPl5LI5+MvsM2WVm5pRTCfBt2xCx96Y6F3Iv3E0NAefUVAoinudJwoXqj58/f/z8WWvd952B0NzdlcLAMQIQBIkCQliQ6RPfpUpEjBQBs/VwF3TtrQWM4wiEjFUQEJGYIcB7X37zV+tHGwHAkpKk6/Waa2OW8fExA9DAXY3c1WEFSW/AxKr9aafNOXpfsCRJWYcTEhHlXADo7eu37XInEXPvGmFA7oTIKYUDI2VJgMxJNOATZQ8QgCQ15Swp3yFPxY+j//XjKb/dE18mwgAGFkNRcN4yWe6R52ltxHP4hJQuJTNzrrXSislj5gh3j1UxmpG7pyQiSXWe5wEAa362JmdjTEQEwJTSvu+IPIdDIMIiJHlrrfeWUorwRX9CouFq09rstRRhRsTr9caIMbXZkURSSnmvYwUduouGqVtNe5ZPQce+76sv6WO4eaipnSnAe8tMAcDhl1oRMZcqKfWpCCAiQdTTkFwQcEwFpFyrqfeplYVEwp3BkR0AAXGMGQA5l/M8Z5+Iv4LEWDAX5GhDJG/7/vblS912yYwAEQoIvtjPwgxESEmyWSBJZrE5INzCLRC4AktgLhunfJ7Px7/+fLep14z/eLuSVKZknIZNZTms//z+Ouc7YAqifLkDy5imROaKQCJExKqwLIxEGAFmvqJLVj++BitLYJhSqrWoWmunmSFCzlsEq84xBiC21nqftW6llDk7ACCSzjWv5svtSkg6ZqmlSHaz0bu2FiIcCNXDjAkJk3w8n733XGtlWjO6um1LTN6Oo48BURLKvuU5ZpI65zxbz7Wo2mhnYt5T8nAWcYC6bff7F3d/PJ6IMPpQtaUxFxEhULQFiTY1D7Wheb+4jt7PlBJTEcbvPz8InATm8xitiTCLqLsHYUqMTAAKAczEyZ0YEUAiFElSyiNizNl7OxsPiKDYmGst97evrvb9/fH+830X1P/4d8l7zcko2tAx4M/neH9/HK2V/Vb3y4Y5pgfyZavt9d10igiA/7ISmsWyCiOOtUVaJwdA+HenSJRSkpzVPRZCyD8nLyuALVZ2FcC6kM0sEM0YSq5J5Hg+Px4fs4+YVkq57hdTdTVTteczIqQUEZJfotWlG1+DH1VdgUVEZKpznOhISKnmrRbTie41J/NgBBY8Xv1xnoc5IvScW5+qysIQBuD/+PbbVDter7IVBx2mzGvcFzWn0Y6cxHIyVZszmDOFj5FSliKjnRjw5X6fc3R0wqhZJKWaCiCOMQRJcpl9TjMfUx0dEBzndKBsFjD0228XMxfhb7//Y5xHf72a6b++P1DK/Xadeow555jvz9Y1vv3z/yZO67mJCMnp6POWUuKVFuYA+KkyRFwTst56TtndiOqiCu37vhRsEXC/30up53mO0RFmyRdPMedYpQ0CusecCgBzagRcrlfEmGO088SI2/XauaFHEsGIRSdMkmJqEhELn00kZ4sYc+hTF3r68Xh8SnQ8IhwZIAIJtr2IpDH1y9tNUlEzVTf00c/Rz7OPSAmZvv/5XyI5lTLaCRZ7KcR8tq6qFBakYINw5WI4GNTEYZ4uNacUHs/nu01jiCrUz2mjf7ldak6E4IvAb+DgwIlZANg/wa9AlIhSAAbAKp8y1rvcCGDqiIhSc6gS1OtewQNC/3h/vh/tuvFKnblcv7x9EUcw91wKu/feCaOUxKZM0sZQNWbRqQ3aMsQQUUoFACNwBam7x6ecHlnVeh/rhFTNtI0ezCySImIM7b1VqETLjJFyToRIEUTkqnNMWCmYZu5OiECYctnqpueJABzgAPLxfCSRum28whiJhXmMMftASTnnWmo/jwj79u03SfnPv/4kJBIAdwsdUxfZ6Jr3VHeI+Pnx4TbQKLHUkpBY5wSfey2Xy3aOJxmJUBYxMwzMSbqeppZr2fZShObQ5+MjxgmzJ7R/+/a1ZgLXQHQMcASM3vti2gkuMgoHAACN8Knau04NJ8NkJWfVlnK6brtOHZmFZU49j9e0QAc1r9sVWt/27X6/9zGGzlxyH2NOR9CwzsyZeZqaUSl5jHkcxyoaFxjqr7++5yycSkT/pcMv5VPvtO45RIKApewmp5zy/f6muoswMQF8Ct3cYyUkB5AjIRMWfP/4AIAsCREdYIbrGAgQi/y+7RszMxETJ5Es6Xi9Ru8EeL1eF8Ri22vr7WhHWY7RcA1vvZ1tLPFdyiXlEohmdrvUj4/HbGfZ6jhHSmn04XMaem8+x8unuutQMlUI8CngEWbzCAFLCB8fP64lC9j//1//+f7Xn//jv/1fRRB8BgAxEQlRTHMAQEBHsJVXjeQBMc0szuOcYZoEfDKncE/oQq42w6fUlFKVxO4giFfB337/9v3Hj+fz43whEmJ4Pz5aa0jEyHMcKsXVVZ1IEBnRcq45V5Ec8fmeMXOEuQUEusUi3COQWzBzyTVJgmCRihhrX2++wAtz9VfEuIgaX69XMxtnW9jDNoaIhAgmTpKQ0DDkUsOsq4ZOud1u53kerVFAydlE39/fCem674y0uBwi4K4fH434FRAA1OY4W5tTJSXJCQjUOq14ssS3y2Ye4IaAJYkQWclAZDpiDiFiDBvdVUWEHOpWCQoAhM2pet+373/98a///H9jnP/x7//893/7nRBCp2JAkCm5BUgmZgxwizkVA4XFECMQADXCzPOeuDCjA4PbaOfUOTEAMUqWXLMaYvi98FbydSv9IJ0DINSmufrUXFIRDBQf/jpeAZFzPs8zIkopzLyCDxbhi2jxW9ISVSxk9+PxWAF6C3UyRn8++7bVWou7j76CoCmly1ozLchcP84+ejuOcGeklZ6HRA4AjMBMREFuc3XKIn9+/6u3jgC1rLKzA6IIzz7+9fFAxH3bPj5eSIBC3qeZPY9m7hHIIpfbdb/sc2o7j5JyEj7Odqm1z/nz/SOXcjyfffS1pXebAJZEhPAcCqGJRQTBzcEyCxG668fPd/B5u16+3f9x32hPf6P/P+1bGGGcC/HGq74HXDNbMweElWbpCNtWI7PpWMpc98hFRPJUPc5DcpWcBdJxPI/jcPdtK2Y6Ri8pEWfVCRhZqJRyuo4xUk7MfByHiKzRzC/A+r7vSzf6963oc04R2fettb48KO6ectpqMYsxhrv+8nnf7/cxBkDsl50AecxnQBQn4VXOlm3T8N77MBUREs6JjMAiCEP0OWJqrpUUz9dHROQkr+OYvRNhKeUIex0/tm2TSH0MRKyJHLirMVNAnP3ctstt38/H43g8ci41sfXzmqhk7q3drpWJHs+PTFBQK+dlQjfGW70+n08pdLvfEF0ED0D1PwDHf/uPy31nPT98nIyRWSAQXQpSAIT2TFOyOBATB3Ezx1QM6fE6LN2ESLEgkKOYWgQkSVP9OF9bqUyRwc6PnwNCw81t27Y59fl4EPNt28ec7lxKAUxjUj9eieK+16lzni8q5XK5zNECckIYru35gVWMAlxNzcwwYhqtcrZkfrtfVefzdQDONl6IkHNOBSy4VDnbcRxHRHj4Vrc59f04U07IfJ5PA5BUQieCb3VvrTFTrr+L5DnHcZziDjnnLKJmfSgTpMseEQggTCxsqstd5mbhrgFjGiDV/XK93eq2O+Lsw2MwUUrJTJ8fHwGQcz5erxVW0lQRMNw4oakxeckFANdxllKYqPXTncyMEVKWKsKhQ8ccJxGNOYUZMY7xyKVwrmHNhkyHgEjbhYl84aHcwx2J55hqVnJZJXd4EHNlZlgDM4QIdzPw1WtFRC6F/o/EdDNTJRFMIuV+R0Q3u9/vv3RNbgYA27blnB/9mK5brSknM5u9E+JWNyISZDMVkdv9erZOhNu2jTnmnPu+RcTj8Rhj7JediFpvdo6VsgCIqeTAZd3B13EsrcYcE4+zVEDEWoqoKaIMM3CXlBLjX399LzldLhchPF6vHz9/3t52yTkicqmAxMl6n2st0lqr2y45W2sMmHPuvavZGvg+Hg8iWnLKfd97P4k4AFTVNBB5jLFKq4ggQhGZk3Scl2slsH4c7fno58dAsKm17Enq6Hq7fynBEz3likBhmLBu2+YgTa0THapmjlA5SZEUEDZVkuSUCWD0AbHQBciS9iREdBwHAJRSWmvrAlwwxN47EbFOJloMtuv1uv64vtfyh44xmAglTVVyl2VRW2Z6QHM7jiOn5O6Pj4eqrpDE1nqStPDtkPKl7jml8zjUdKt1zmF/s8Mej8e+7+uf5pLP1no/EVGEiUi+fv16HEdrfWHFj7MhIhGPMR69Q8S333//+vXt7OfrOCLAQnsfxELMah7qIspJumprnwFHgDDn9IiUEovUbWu9E1EAEMtWd9WYswvTEv/MOSJMUmaR1+t07TVdyO04XjpajKGh7TjtbMzl7NOm/vYPSGKUgTjFjPGiknKuCUFmyQeTapSccL8wkc4JAIws+Bkjsl44Id5qLfK/xZ8LaPHXX38tStcvPCkTHr3lnDmnYUpJ5pyzaR9DTXcmFkGKYXO2EwEg59WB9NaVNYv41PP5MjMA8qH/+ef/V0v57ds3IuqvQ4CIQPuwMRdHMyLWHA8RzT99ghERhCnnAjCHu3YHIRF5nX2MMcfY973UCoCzt7U+KaWw8PVyDeKjjT6s7hchQmRYz7EkZnH3/ny5OzJFxFqNOkx1Q+FAmKZA2FWRuc+ZcwxVdU/EyAnZA3xGYESf+v395+9J3q7X0Hba3EpC2UEnqY0xfToa9uP148/xeMj97evbl3/kek0cNs8+J/BemCvLBM8pK8Dow80gwObs7oSIAMIS5oCYc27HY5kffqEQlublV5HSWoMk4R5/J8gsXdMSGKacx5xkBkJIeLtecymE2M9mAcLExIzUtc8551TmdL/dRVIthUk+Hh+ElJIgoE4jJkKec7Yx14M1pyJRKXVOPXpzxN6HiCQhBg+byCjv7+8pZ0mFWIiYUyJCnXPOtm1lq1tO6Rjz6IYo+35lZnMw97N3ca+1LvNAzZkzrHBxHWONhlctfhzHgh/fbrfezwA0i2UBHWOY+bYVYkAIJhJOBctWaz+Gu9WcSk2znbGZ6wtQ7vc7AE1/vT4epkpIFw8PVg+DJIWCkptqVx7DhE1VmIXYzEbvpRRhFmY3c11uMDjP89dSd0Hz1vnFZ0pLPF/Pr1++IFFv5/V2G72b+9lOM0859dERiZ0Bg0oRIiYeiIiYpeQVgNUaIn55ewtHnSrEs4/mDTw4kam52crsdLMfTz2Po9QiScacWSQCHOF2v+/75fl8jjnfrjeMGHP03uTt61dG6r0/nq/e++162a8XVT1eDyKpdRtzHn2mvCFCn0YWgQyIueQI6H0QEQCqquSERJ/MHgBTRaKvv/3WW3s8n/fbvfVBjI/XM0sNwKF6vdwlOTKqzbe3m+q4XK/x+K4eQx0wlVoFFMRecO7X+2W/BzJBMuDv75MYc80OrjY5Qk0NGzJ6hLmN0S0JeEwdsm3bti0xZk7JzY7ni5l1zqVt+XznAC6XyyKnI+JxHMuvRGA5pcfj4RGv18vMxpwppb1Wd19hAaZKBKb68f6xBhrC7IBGNMf4W3WIqv7z/UGEtVYiQRIPF07H2ft4qvntdpfUtwuomlsQiwOOPiQJM7XWI9A9VCf8vduXZY/ilGj0MfVsI6dca3X31trPjw83e3RnSbWWILYI5uQ+FwwnIrYszCvJbNEnAYhi3eNTV4GwbtrZ5jw7IiFMxLTVnCSd7WTOTJFScleW9P5sfQJI2W9f3367tud7AG9XFMmX/T6nRxBacHpK2vbrW95uKBdOe837MPnr+8f0yPumgGCeRNZKqbfWW9vqBpLCXERWaI3O1ntfQ7KlGVxI4GWoj4haq/CnuoBF1kciQszmvm6dT9UuMyGFOyKWlMLjE8WsyoCCxEh8qW+IZzuPsxHTtm3gRCKSk059Hsd0W5RsQFwzWXVnkW3fL9crMR2tnef58XotppS5y3m2nDMzpVyWfuL98Zha5hjP5/NkLjnlcnHkNhzGmn2Ah1GAmQOoIEWAJFroJEqCgAunAwIe4RCcJABIuKa91nq8+krrO/pAYk5J1duYZ2tjTof885iJIOWL84bZhMstXUrZa9nPc051H5a3L+Vyk3qXekXZDFJg6m6vOSmny/UNSjGWdp69dzdLOe/bXktNzKM1RAy3H9/fmWPprJdIcE1eXq/X6/VaF6y7H70lkTnnZ9OyhExmq49iZlsE+nWeAQTExEgrKQeFGDLlnByxzVkuO5U8fvzwCClFzZqpI6Z9KzlLSr33zBKIptp1uup22XOpxELCpUAEKJOwRMQ8TyFmjwBzJBIWJjKdY6iZl1JqraUUk/2c1s8TkRCiH+e21brX2ZuOpbHzPub0AQFr/ebha6YsJU3T1tqP95/ElEtmSYGdhHMpK00BiYYq9OhTA5C22399f/7+7X57+3L6SOXC4UK8bRfCNKDbsNBWbnh9+43y1XnjtInUj1f7+TioFA18HycvJc4nqQn62YiocHqe53meW63Lnavafp3N2iuVUlZH+EsyUxITkUfQ37reX17tXwJtcAgPC1vX6URMIhERap/JJtMgMaU8I4CZRFrv55yI+Hwd6/12gBxrW8yEGALkDuRrbaJqiVg414oHwDQDxHq7yfV6XfZGhFBznZOJphkh1H1PktztdfZA2S7XlJKONscwc1MzczObM8wMGYskD//VMgNAkpQkadL1q3O73RhZ1QGQkOacqlZyVbM5ZimSUy6lHpRb618wbfdvqAe5hs1t22q9qAZbYrQUk+Fab1/S/gUlAedz2H/9eH+1eX37gsgWwITouKgxiFhy3uvGzK/Xc6mbwLym5PnzPH6ZktaYekWmfa6fEgat3Ef71U0RERAZhJoupHgEICIC+JoXrNGhu6QkxG7qiArx8/v3XIoTSslAOObkLPWyEVEuhQDn2db7bvHZF72/v3NK27YRs0Oc5+nEwcTM++Uiv2zHARhmc06bo+T85e22yoExbejgLBD0fD51jLpfYs7WGiNKkpoSEXKispf1/f+m6sIq/1R1cZO2bVOdYzRmLnljFggyNXPVaRFoqs/nc7/cEfmP7+/bVv77//NPHwdY2q73mmtrypME7cslzXECC6WCnL+/P348ntMCmJ/tvL59yylRgLBHeEopsZRSbtfbAjhnEZ3z1fr1dl0U7dVjrGbj+XwyL4XsZyjJ6/hYqrWUMi8575wAQYhzTndLObM7MCVhQDKylCQ85hiIkCAnkRkx5vw4zjbHdrkg0ePxmKorIWqauscesdeakpjadIMISZKZ5xzTrPeeSs457/s+A4EwSZIk0o9T1cx027bpQciYyzD/8/1R+8w5BwkAMIX5NFcQnK7DBgufOkvOhhChN9mt6+idiLctteEOyYIQ8eyNyHO9OqYe4BEUeNmqmh/j7NYjbMZ8f39XV2LZLhcLfz3n//zXDy63395uRJDwAlysBOGg1gfThz0ej+f//Phzqplbqdvbl98cYNv3n+8ficknkMlte9u2EmFHa/86D2K21Q3X/TzaMQNIdU4WQus65vDe7RCURFXNus5sRY04JRTCnMtln6rj8UgpOXHEBHRVQJaQ1B0AXHLubmO01loSLrfbGf48D3fIkHLeN+DhUTk7wuVy890t4PF8UKpKwpwcDcxTEkTsvVO6vN0qfN78klPq5wkGlIQxSWJh4udzuLm7P57P3779tl8uH8/nq/W87QBg3kyVmXNJJHIeBzJxTkc/Z3czCzD3qMR127dtY+bjHCljAJmFOZgHS7yOni8Xd9cxLWCaOgQKJeJtr+GhC5VFSCLb5daO88+fL4PUjteen/f7dSsli2Cuw3zyhgVNdWpDTnW/kxRw//OP7wjQ4qxyYcqMqNPcJwBMm+08TM0Cb1ehVIgp3EggF/aw0UegT5uBIMkBWN1hauKcchUEQELJEcF1c4CjDx8zS4KA1tUCidDckwcCADGmNKa+P5/mhgFb2WZzndqpsaRL2Yfr8/Fsc+6XqwOdrVfks/VS68JxB0SQAFIfWmtV1e/f30vOlJO5z6EQKO8fH2uKGPiJrOq9397u21bVNFYi+qdqFHNKRIvIKsvR4e7ILJKy5EQSDsfrjAi3qHV3dzfNqSKSm5trIWJEZzOzgNi2TYRtDlUlJEQkZGLJpaaUI9ADnq/z+fHxAHx/PIowsyThcr2jyL7tROS3TxzFx8eDiP/4489//vOfo820+75ldz1eRxsnMizdaEqZDHRY64OZwXvOEkE6pykA4r5dRVKtGwSPYYgYAGsFZma9taEzr19NC5KcRXrrfycqM4SNoSuMZ6tb5JUz77mWfbs0nHaegBQAXYdFeMQYM1cl4uNsJEmIS84RMMZABGHZ9+35fLqHiKhaxEgA5kaIa1IovfeUMyL20X//x+9m9vF4lJxvt9uYc9EBlkwLPiMd3F3nVKbP3oiJjvPojmtbpqo/f/7c6rZfdvg7H/fvKaUlEUnZIeZQzEKUurvNSUTMyMJOAhzEKMWllJRkv9xNJ7I4klsQcwCEWZ9zVSWllNfr9fHxERFfvnxR1UDovWcaRGAAvzS3NrzWipwQxJHDY8wzg7ihKniAG+S8r8ELAK215b5lJPYIg3BzRGJOCMAbF0mJxczhk67vxCnQzLT3SYTCJJIQccz58Jdw3fY9EI92Hq2TMKf8j3+7ApJ6kAxEvOxXBBqzmzkzA6A71HpZ7Orb7X4chwMG0BjKPAVZkHVlAATg16/fhur3H99TrqXWP/74AxklZ1ADAJ1zzOFmOWWIkMTHccw5LPw8GwMRMecEhEAYBGulycyxorxF5pwAmIQDaKoCgKol4XzJBKA65xyhYEYiBJQoVclZUnWdtZQsrHOWUkh4ztF7P89zrSBKKV+/fv3x48fb25fH4+N6u64E98CQSB7J3IXherubmZqVUq7XWxIZfUtJVIcjBuJ5tmmwCP8pZRbiwJzS0mgx5yA0N05Z53w9z4btsu1LRzLm9M8tBIEDIDh462NMTcLu+BonxKy1AlEbExAdsJ8tl8083H2rm6oPU3cfvQMiCkO4QeyX/TgPj7hebuoWiCtOq48uS3f8fL04pd/f3s7eztZyzpzSUA3CXxnFS3DQe9tKIcQxJwuFeeZ0vV3AIADXQERVS92+fP26SHQp5VJE5wwk5qyfMeG0cgZa7yldRZLO6QEpFSdh4pwzAJFkQzY1CCQpUqRPUwOfPcKZhEmWg5yQucrb/cvz8cypzK4oWSnmGEP7IkibG0nqfbpjrUJClNI1FSQYvUnK2z5FnmbaxyTOIoUFCSVw6Jyp5G3fR+j5/sGciOSgNsYUVvAIYk6YiIVp9EbMiCCcZdtN1U05C6Xcu75GR0RgSpIkpWjjzz//un35EgHLKTPH3PadeSkKluEteUTOOTymWspF3ZgFdJqZREDKJc3JtCibYR7M3Prw8JRySnlOVdXeBwuamokhwuPjI3QDj1ILmDGzeuRSaq2v12sN8JbHziNKzu4OSCFsw1RNRJDYAEqpKaXFogekQCJOJOgRJOIRauYBOvXn+/u2bYLYx3Sbc/bVCQDAksO01r5+/fp6PVNK7kZCvY/n60ECKYtIISFVP8729uXr7X4bYzLL979+JpEAZ8Y+dDp4UNkupnEOLbmqQxEulYcqimibKRckTsxfvnwdvY/jJCSRjIRm6gAOPLXdLpd2nul6DXUFtD5FkhOZWU6JEEvdiNmRh1qt+9QZAbfbHRz2y6X3rmpzzm3bzrOJpJQkArXPiEi5SEp2xuhDWh8e0MZgSdH5y9cvM3zMaWMCIQIuhae5jTErJTdT1VpymNlUIvKpP96fUmrKVdXONswjl23lk+dciVgkmTkSKzKzmCsu5qB72TKxRDgxh+McUyQH4uidCcMDEJOIjXmeJxHLVh+Pn7XkOQ2R3MHdmIU5IaJZpFSYhYi2und0OIA5MTMxlLox8/M4icUjnseZc2ljPo9DdRDD1J5zgghG7taFmEs5XmfYvFwvaNi6vp6tbBWA5wxHrttF20JrCKZlxuVceYyRSv14PIk51ZqhttYCYvTOLJxLREhKgBwwr7d7qRU6msX1elfTUisgqJn5k1PqrdctsQgAyOLesiyoLLP8L4S47qLEeCHYAAAAAElFTkSuQmCC\n", "text/plain": [ - "" + "" ] }, - "execution_count": null, + "execution_count": 4, "metadata": {}, "output_type": "execute_result" } @@ -656,13 +712,13 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { - "model_id": "71f795aa7c744dedb6edf0304ad6ab3d", + "model_id": "396649734fe5456893208bc29eb1d8db", "version_major": 2, "version_minor": 0 }, @@ -736,6 +792,13 @@ "print(f\"Is this a cat?: {is_cat}; Probability it's a cat: {probs[1].item():.6f}\")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now let's take a step back and have a look at what we actually did when running those lines of code." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -749,9 +812,9 @@ "source": [ "Well that was impressive--we trained a model! But... what does that actually *mean*? What did we actually *do*?\n", "\n", - "To answer those questions, we need to step up a level from *deep learning* and discuss the more general *machine learning*. *Machine learning* is (like regular coding) a way to get computers to do stuff. But how would you use regular coding to do what we just did in the last section: recognize dogs vs cats in photos? We would have to write down for the computer the exact steps necessary to complete the task.\n", + "To answer those questions, we need to step up a level from *deep learning* and discuss the more general *machine learning*. *Machine learning* is (like regular coding) a way to get computers to complete a specific task. But how would you use regular coding to do what we just did in the last section: recognize dogs vs cats in photos? We would have to write down for the computer the exact steps necessary to complete the task.\n", "\n", - "Normally, it's easy enough for us to write down the steps to complete a task when we're writing a program. We just think about the steps we'd take if we had to do the task by hand, and then we translate them into code. For instance, we can write a function that sorts a list. In general, we write a function that looks something like this (where *inputs* might be an unsorted list, and *results* a sorted list):" + "Normally, it's easy enough for us to write down the steps to complete a task when we're writing a program. We just think about the steps we'd take if we had to do the task by hand, and then we translate them into code. For instance, we can write a function that sorts a list. In general, we write a function that looks something like <> (where *inputs* might be an unsorted list, and *results* a sorted list)." ] }, { @@ -1201,7 +1264,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### A bit of deep learning jargon" + "#### A bit of deep learning jargon" ] }, { @@ -1366,6 +1429,13 @@ "This can also create problems in commercial products. For instance, a video recommendation system might be biased towards recommending content consumed by the biggest watchers of video (for instance, conspiracy theorists and extremists tend to watch more online video content than average), resulting in those users increasing their video consumption, resulting in more of those kinds of videos being recommended..." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now that we have seen the base of the theory, let's go back to our code example and see how the code corresponds to the process we just described." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1426,6 +1496,8 @@ "\n", "The other important piece of information that we have to tell fastai is how to get the labels from the dataset. Computer vision datasets are normally structured in such a way that the label for an image is part of the file name, or path, most commonly the parent folder name. Fastai comes with a number of standardized labelling methods, and ways to write your own. Here we define a function `is_cat` which labels cats based on a filename rule provided by the dataset creators.\n", "\n", + "TK Sylvain. Check conversion here, there is a problem with formatting\n", + "\n", "Finally, we define the `Transform`s that we need. A `Transform` contains code that is applied automatically during training; fastai includes many pre-defined `Transform`s, and adding new ones is as simple as creating a Python function. There are two kinds: `item_tfms` are applied to each item (in this case, each item is resized to a 224 pixel square); `batch_tfms` are applied to a *batch* of items at a time using the GPU, so they're particularly fast (we'll see many examples of these throughout this book).\n", "\n", "Why 224 pixels? This is the standard size for historical reasons (old pretrained models require this size exactly), but you can pass pretty much anything. If you increase the size, you'll often get a model with better results (since it will be able to focus on more details) but at the price of speed and memory consumption; or visa versa if you decrease the size. " @@ -1464,7 +1536,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**Overfitting is the most important and challenging single issue** when training for all machine learning practitioners, and all algorithms. As we will see, it is very easy to create a model that does a great job at making predictions on the exact data which it has been trained on, but it is much harder to make predictions on data that it has never seen before. And of course this is the data that will actually matter in practice. For instance, if you create and hand-written digit classifier (as we will very soon!) and use it to recognise numbers written on cheques, then you are never going to see any of the numbers that the model was trained on -- every cheque will have slightly different variations of writing to deal with. We will learn many methods to avoid overfitting in this book. However, you should only use those methods after you have confirmed that overfitting is actually occurring (i.e. you have actually observed the validation accuracy getting worse during training). We often see practitioners using over-fitting avoidance techniques even when they have enough data that they didn't need to do so, ending up with a model that could be less accurate than what they could have gotten." + "**Overfitting is the most important and challenging single issue** when training for all machine learning practitioners, and all algorithms. As we will see, it is very easy to create a model that does a great job at making predictions on the exact data which it has been trained on, but it is much harder to make predictions on data that it has never seen before. And of course this is the data that will actually matter in practice. For instance, if you create a hand-written digit classifier (as we will very soon!) and use it to recognise numbers written on cheques, then you are never going to see any of the numbers that the model was trained on -- every cheque will have slightly different variations of writing to deal with. We will learn many methods to avoid overfitting in this book. However, you should only use those methods after you have confirmed that overfitting is actually occurring (i.e. you have actually observed the validation accuracy getting worse during training). We often see practitioners using over-fitting avoidance techniques even when they have enough data that they didn't need to do so, ending up with a model that could be less accurate than what they could have gotten." ] }, { @@ -1484,7 +1556,7 @@ "\n", "The fourth line tells fastai to create a *convolutional neural network* (CNN), and selects what *architecture* to use (i.e. what kind of model to create), what data we want to train it on, and what *metric* to use. A CNN is the current state of the art approach to creating computer vision models. We'll be learning all about how they work in this book. Their structure is inspired by how the human vision system works.\n", "\n", - "There are many different *architectures* in fastai, which we will be learning about in this book, as well as discussing how to create your own. Most of the time, however, picking an architecture isn't a very important part of the deep learning process. It's something that academics love to talk about, but in practice it is unlikely to be something you need to spend much time on. There are some standard architectures that work most of the time, and in this case we're using one called _ResNet_ that will be learning a lot about during the book, and is both fast and accurate for many datasets and problems. The \"34\" in `resnet34` refers to the number of layers in this variant of the architecture (other options are \"18\", \"50\", \"101\", and \"152\"). Models using architectures with more layers take longer to train, and are more prone to overfitting (i.e. you can't train them for as many epochs before the accuracy on the validation set starts getting worse). On the other hand, when using more data, they can be quite a bit more accurate.\n", + "There are many different architectures in fastai, which we will be learning about in this book, as well as discussing how to create your own. Most of the time, however, picking an architecture isn't a very important part of the deep learning process. It's something that academics love to talk about, but in practice it is unlikely to be something you need to spend much time on. There are some standard architectures that work most of the time, and in this case we're using one called _ResNet_ that will be learning a lot about during the book, and is both fast and accurate for many datasets and problems. The \"34\" in `resnet34` refers to the number of layers in this variant of the architecture (other options are \"18\", \"50\", \"101\", and \"152\"). Models using architectures with more layers take longer to train, and are more prone to overfitting (i.e. you can't train them for as many epochs before the accuracy on the validation set starts getting worse). On the other hand, when using more data, they can be quite a bit more accurate.\n", "\n", "A *metric* is a function that is called to measure how good the model is, using the validation set, and will be printed at the end of each *epoch*. In this case, we're using `error_rate`, which is a function provided by fastai which does just what it says: tells you what percentage of images in the validation set are being classified incorrectly. Another common metric for classification is `accuracy` (which is just `1.0 - error_rate`). fastai provides many more, which will be discussed throughout this book." ] @@ -1527,6 +1599,13 @@ "> jargon: Metric and Loss: A *metric* is a calculation that is made after each epoch and displayed so that you can see how well your model is training. It's not used as part of the actual learning process. The *loss* is the \"measure of performance\" that is used by the learning process to define whether one set of parameters is better or worse than another; the learning process works to make the loss as low as possible." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "So, with all this code our model learned to recognize cats and dogs just from labeled examples. But how did it do it?" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1538,7 +1617,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "At this stage we have an image recogniser that is working very well, but we have no idea what it is actually doing! Although many people complain that deep learning results in impenetrable \"black box\" models, this really couldn't be further from the truth. There is a vast body of research showing how to deeply inspect deep learning models, and get rich insights from them.\n", + "At this stage we have an image recogniser that is working very well, but we have no idea what it is actually doing! Although many people complain that deep learning results in impenetrable \"black box\" models (that is, something that gives predictions but that no one can understand), this really couldn't be further from the truth. There is a vast body of research showing how to deeply inspect deep learning models, and get rich insights from them.\n", "\n", "In 2013 a PhD student, Matt Zeiler, and his supervisor, Rob Fergus, published the paper [Visualizing and Understanding Convolutional Networks](https://arxiv.org/pdf/1311.2901.pdf), which showed how to visualise the neural network weights learned in each layer of a model. They carefully analysed the model that won the 2012 ImageNet competition, and used this analysis to greatly improve the model, such that they were able to go on to win the 2013 competition! Here is the picture that they published of the first two layers' weights:" ] @@ -1586,7 +1665,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "This article was studying an older model called `AlexNet` that only contained five layers. Networks developed since then can have hundreds of layers--so you can imagine how rich the features developed by these models can be!" + "This article was studying an older model called `AlexNet` that only contained five layers. Networks developed since then can have hundreds of layers--so you can imagine how rich the features developed by these models can be! \n", + "\n", + "When we fine-tuned our pretrained model earlier, we adapted what those last layers focus on (flowers, humans, animals) to specialize on the cats versus dogs problem. More generally, we could specialize such a pretrained problem on many different tasks. Let's have a look at some examples. " ] }, { @@ -1616,7 +1697,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Time series can be easily converted into an image by simply plotting the time series in the usual way. However, it is often a good idea to try to represent your data in a way that makes it as easy as possible to pull out the most important components. In a time-series, things like seasonality and anomalies are most likely to be of interest. There are various transformations available for time series data; for instance, fast.ai student Ignacio Oguiza created images from a time series data set for olive oil classification. He used a technique called Gramian Angular Field (GAF), and then fed those images to an image classification model just like the one you see in this chapter. His results, despite having only 30 training set images, were well over 90% accurate, and close to the state-of-the-art." + "Time series can be easily converted into an image by simply plotting the time series in a graph. However, it is often a good idea to try to represent your data in a way that makes it as easy as possible to pull out the most important components. In a time-series, things like seasonality and anomalies are most likely to be of interest. There are various transformations available for time series data; for instance, fast.ai student Ignacio Oguiza created images from a time series data set for olive oil classification. He used a technique called Gramian Angular Field (GAF), and you can see the result in <>. He then fed those images to an image classification model just like the one you see in this chapter. His results, despite having only 30 training set images, were well over 90% accurate, and close to the state-of-the-art." ] }, { @@ -1630,7 +1711,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Another interesting fast.ai student project example comes from Gleb Esman. He was working on fraud detection at Splunk, and was working with a dataset of users' mouse movements and mouse clicks. He turned these into pictures by drawing an image where the position, speed and acceleration of the mouse was displayed using coloured lines, and the clicks were displayed using [small coloured circles](https://www.splunk.com/en_us/blog/security/deep-learning-with-splunk-and-tensorflow-for-security-catching-the-fraudster-in-neural-networks-with-behavioral-biometrics.html). He then fed this into an image recognition model just like the one we've shown in this chapter, and it worked so well that had led to a patent for this approach to fraud analytics!" + "Another interesting fast.ai student project example comes from Gleb Esman. He was working on fraud detection at Splunk, and was working with a dataset of users' mouse movements and mouse clicks. He turned these into pictures by drawing an image where the position, speed and acceleration of the mouse was displayed using coloured lines, and the clicks were displayed using [small coloured circles](https://www.splunk.com/en_us/blog/security/deep-learning-with-splunk-and-tensorflow-for-security-catching-the-fraudster-in-neural-networks-with-behavioral-biometrics.html) as shown in <>. He then fed this into an image recognition model just like the one we've shown in this chapter, and it worked so well that had led to a patent for this approach to fraud analytics!" ] }, { @@ -1644,7 +1725,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Another examples comes from the paper [Malware Classification with Deep Convolutional Neural Networks](https://ieeexplore.ieee.org/abstract/document/8328749) which explains that \"the malware binary file isdivided into 8-bit sequences which are then converted to equivalent decimalvalues. This decimal vector is reshaped and gray-scale image is generated thatrepresent the malware sample\":" + "Another examples comes from the paper [Malware Classification with Deep Convolutional Neural Networks](https://ieeexplore.ieee.org/abstract/document/8328749) which explains that \"the malware binary file isdivided into 8-bit sequences which are then converted to equivalent decimalvalues. This decimal vector is reshaped and gray-scale image is generated that represent the malware sample\", like in <>" ] }, { @@ -1658,7 +1739,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "They then show \"pictures\" generated through this process of malware in different categories:" + "They then show \"pictures\" generated through this process of malware in different categories, as shown in <>." ] }, { @@ -1681,14 +1762,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### A bit more jargon" + "### Jargon recap" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Here's a summary of the deep learning vocabulary we've met so far:\n", + "We have introduced quite a few new terms, <> contains a handy recap of most of them.\n", "\n", "```asciidoc\n", "[[dljargon]]\n", @@ -1726,7 +1807,7 @@ "\n", "In order to make the training process go faster, we might start with a *pretrained model*, a model which has already been trained on someone else's data. We then adapt it to our data by training it a bit more on our data, a process called *fine tuning*.\n", "\n", - "When we train a model, a key concern is to ensure that our model *generalizes* -- that is, that it learns general lessons from our data which also apply to new items it will encounter, so that it can make good predictions on those items. The risk is that if we train our model badly, instead of learning general lessons it effectively memorizes what it has already seen, and then it will make poor predictions about new images. Such a failure is called *overfitting*. In order to avoid this, we always divide our data into two parts, the *training set* and the *validation set*. We train the model by showing it only the *training set* and then we evaluate how well the model is doing by seeing how well it predicts on items from the *validation set* . In this way, we check if the the lessons the model learns from the training set are lessons that generalize to the validation set. In order to assess how well the model is doing on the validation set overall, we define a *metric* . During the training process, when the model has seen every item in the training set, we call that an *epoch* .\n", + "When we train a model, a key concern is to ensure that our model *generalizes* -- that is, that it learns general lessons from our data which also apply to new items it will encounter, so that it can make good predictions on those items. The risk is that if we train our model badly, instead of learning general lessons it effectively memorizes what it has already seen, and then it will make poor predictions about new images. Such a failure is called *overfitting*. In order to avoid this, we always divide our data into two parts, the *training set* and the *validation set*. We train the model by showing it only the *training set* and then we evaluate how well the model is doing by seeing how well it predicts on items from the *validation set* . In this way, we check if the lessons the model learns from the training set are lessons that generalize to the validation set. In order to assess how well the model is doing on the validation set overall, we define a *metric* . During the training process, when the model has seen every item in the training set, we call that an *epoch* .\n", "\n", "All these concepts apply to machine learning in general. That is, they apply to all sorts of schemes for defining a model by training it with data. What makes deep learning distinctive is a particular class of architectures, the architectures based on *neural networks*. In particular, tasks like image classification rely heavily on *convolutional neural networks*, which we will discuss shortly." ] @@ -1869,7 +1950,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We are not even going to walk through this code line by line, because it is nearly identical to our previous example! (Although we will, of course, be doing a deep dive into segmentation models later in the book, along with all of the other models that we are briefly introducing in this chapter, and many, many more.)\n", + "We are not even going to walk through this code line by line, because it is nearly identical to our previous example! (Although we will, of course, be doing a deep dive into segmentation models in <>, along with all of the other models that we are briefly introducing in this chapter, and many, many more.)\n", "\n", "We can visualise how well it achieved its task, by asking the model to color code each pixel of an image. As you can see, it nearly perfectly classifies every pixel in every object; for instance, notice that all of the cars are overlaid with the same colour, and all of the trees are overlaid with the same color (in each pair of images, the left hand image is the ground truth labels, the right hand is the predictions from the model):" ] @@ -2013,8 +2094,6 @@ } ], "source": [ - "#id training2\n", - "#caption Training loop in a text application\n", "from fastai2.text.all import *\n", "\n", "dls = TextDataLoaders.from_folder(untar_data(URLs.IMDB), valid='test')\n", @@ -2063,6 +2142,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "Here we can see the model has considered the review to be positive. The second part of the result is the index of \"pos\" in our data vocabulary and the last part is the probabilities attributed to each class (99.6% for \"pos\" and 0.4% for \"neg\"). \n", + "\n", "Now it's your turn! Write your own mini movie review, or copy one from the Internet, and we can see what this model thinks about it. " ] }, @@ -2089,7 +2170,7 @@ "\n", "The outputs themselves can be deceiving: they have the results of the last time the cell was executed, but if you change the code inside a cell without executing it, you will keep them.\n", "\n", - "Except when we mention it explicitly, the notebooks provided on the book website are meant to be run in order, from top to bottom. In general, when experimenting, you will find yourself executing cells in any order to go fast (which is a super neat feature of Jupyter Notebooks) but once you have explored and arrive at the final version of your code, make sure you can run the cells of your notebooks in order (your future self won't necessarily remember the convoluted path you took otherwise!). \n", + "Except when we mention it explicitely, the notebooks provided on the book website are meant to be run in order, from top to bottom. In general, when experimenting, you will find yourself executing cells in any order to go fast (which is a super neat feature of Jupyter Notebooks) but once you have explored and arrive at the final version of your code, make sure you can run the cells of your notebooks in order (your future self won't necessarily remember the convoluted path you took otherwise!). \n", "\n", "In edit mode, pressing `0` twice will restart the *kernel* (which is the engine powering your notebook). This will wipe your state clean and make it as if you had just started in the notebook. Clean then on the \"Cell\" menu and then on \"Run All Above\" to run all the cells above the point you are. We have found this to be very useful when developing the fastai library." ] @@ -2387,7 +2468,7 @@ "source": [ "This model is predicting movie ratings on a scale of 0.5 to 5.0 to within around 0.6 average error. Since we're predicting a continuous number, rather than a category, we have to tell fastai what range our target has, using the `y_range` parameter.\n", "\n", - "Although we're not actually using a pretrained model (for the same reason that we didn't for the tabular model), this example shows that fastai let's us use `fine_tune` even in this case (we'll learn how and why this works later in the book). We can use the same `show_results` call we saw earlier to view a few examples of user and movie IDs, actual ratings, and predictions:" + "Although we're not actually using a pretrained model (for the same reason that we didn't for the tabular model), this example shows that fastai let's us use `fine_tune` even in this case (we'll learn how and why this works later in <>). We can use the same `show_results` call we saw earlier to view a few examples of user and movie IDs, actual ratings, and predictions:" ] }, { @@ -2533,6 +2614,13 @@ "### End sidebar" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Each of the models we trained showed a training and validation loss. A good validation set is one of the most important piece of your training, let's see why and learn how to create one." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -2587,7 +2675,7 @@ "\n", "One case might be if you are looking at time series data. For a time series, choosing a random subset of the data will be both too easy (you can look at the data both before and after the dates your are trying to predict) and not representative of most business use cases (where you are using historical data to build a model for use in the future). If your data includes the date and you are building a model to use in the future, you will want to choose a continuous section with the latest dates as your validation set (for instance, the last two weeks or last month of the available data).\n", "\n", - "Suppose you want to split the time series data below into training and validation sets:" + "Suppose you want to split the time series data in <> into training and validation sets:" ] }, { @@ -2601,7 +2689,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "A random subset is a poor choice (too easy to fill in the gaps, and not indicative of what you'll need in production):" + "A random subset is a poor choice (too easy to fill in the gaps, and not indicative of what you'll need in production), as we can see in <>" ] }, { @@ -2615,7 +2703,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Use the earlier data as your training set (and the later data for the validation set):" + "Use the earlier data as your training set (and the later data for the validation set), as shown in <>." ] }, { @@ -2666,6 +2754,13 @@ "Sometimes it may not be clear how your validation data will differ. For instance, for a problem using satellite imagery, you'd need to gather more information on whether the training set just contained certain geographic locations, or if it came from geographically scattered data." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now that you have got a taste of how to build a model, you can decide what you want to dig into next." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -2677,7 +2772,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Now that you have got a taste of how to build a model, you can decide what you want to dig into next. If you would like to learn more about how to use deep learning models in practice, including identifying and fixing errors, and creating a real working web application, and how to avoid your model causing unexpected harm to your organization or society more generally, then keep reading the next chapters, _From model to production_, and _Data ethics_. If you would like to start learning the foundations of how deep learning works _under the hood_, skip to <>, _Under the hood: training a digit classifier_. (Did you ever read _Choose Your Own Adventure_ books as a kid? Well, this is kind of like that… except with more deep learning than that book series contained.)\n", + "If you would like to learn more about how to use deep learning models in practice, including identifying and fixing errors, and creating a real working web application, and how to avoid your model causing unexpected harm to your organization or society more generally, then keep reading the next chapters, _From model to production_, and _Data ethics_. If you would like to start learning the foundations of how deep learning works _under the hood_, skip to <>, _Under the hood: training a digit classifier_. (Did you ever read _Choose Your Own Adventure_ books as a kid? Well, this is kind of like that… except with more deep learning than that book series contained.)\n", "\n", "Either way, you will need to read all these chapters in order to progress further in the book; but it is totally up to you which order you read them in. They don't depend on each other. If you skip ahead to <>, then we will remind you at the end of that section to come back and read the chapters you skipped over before you go any further." ] @@ -2707,7 +2802,7 @@ " - A PhD T / F\n", "1. Name five areas where deep learning is now the best in the world\n", "1. What was the name of the first device that was based on the principle of the artificial neuron?\n", - "1. Based on the book of the same name, what are the either requirements for \"Parallel Distributed Processing\"?\n", + "1. Based on the book of the same name, what are the requirements for \"Parallel Distributed Processing\"?\n", "1. What were the two theoretical misunderstandings that held back the field of neural networks?\n", "1. What is a GPU?\n", "1. Open a notebook and execute a cell containing: `1+1`. What happens?\n", diff --git a/02_production.ipynb b/02_production.ipynb index 9984147..0237d68 100644 --- a/02_production.ipynb +++ b/02_production.ipynb @@ -29,7 +29,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The five lines of code we've seen are just one small part of the process of using deep learning in practice. In this section, we're going to use a computer vision example to look at the end-to-end process of creating a deep learning application. More specifically: we're going to build a bear classifier! In the process, we'll discuss the capabilities and constraints of deep learning, learn about how to create datasets, look at possible gotchas when using deep learning in practice, and more." + "The five lines of code we've seen in <> are just one small part of the process of using deep learning in practice. In this chapter, we're going to use a computer vision example to look at the end-to-end process of creating a deep learning application. More specifically: we're going to build a bear classifier! In the process, we'll discuss the capabilities and constraints of deep learning, learn about how to create datasets, look at possible gotchas when using deep learning in practice, and more. Let's start with how you should frame your problem.\n", + "\n", + "TK: the next section title seems a bit inadequate, let's double check" ] }, { @@ -48,117 +50,6 @@ "The best thing to do is to keep an open mind. If you remain open to the possibility that deep learning might solve part of your problem with less data or complexity than you expect, then it is possible to design a process where you can find the specific capabilities and constraints related to your particular problem as you work through the process. This doesn't mean making any risky bets — we will show you how you can gradually roll out models so that they don't create significant risks, and can even backtest them prior to putting them in production." ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### The state of deep learning" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In general, here is a summary of the state of deep learning is at the start of 2020. However, things move very fast, and by the time you read this some of these constraints may no longer exist. We will try to keep the book website up-to-date; in addition, a Google search for \"what can AI do now\" there is likely to provide some up-to-date information." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Computer vision**: there are many domains in which deep learning has not been used to analyse images yet, but those where it has been tried have nearly universally shown that computers can recognise what items are in an image at least as well as people can — even specially trained people, such as radiologists. This is known as *object recognition*. Deep learning is also good at recognizing whereabouts objects in an image are, and can highlight their location and name each found object. This is known as *object detection* (there is also a variant of this we saw in <>, where every pixel is categorized based on what kind of object it is part of--this is called *segmentation*). Deep learning algorithms are generally not good at recognizing images that are significantly different in structure or style to those used to train the model. For instance, if there were no black-and-white images in the training data, the model may well do poorly on black-and-white images. If the training data did not contain hand-drawn images then the model will probably do poorly on hand-drawn images. There is no general way to check what types of image are missing in your training set, but we will show in this chapter some ways to try to recognize when unexpected image types arise in the data when the model is being used in production (this is known as checking for *out of domain* data).\n", - "\n", - "One major challenge for object detection systems is that image labelling can be slow and expensive. There is a lot of work at the moment going into tools to try to make this labelling faster and more easy, and require less handcrafted labels to train accurate object detection models. One approach which is particularly helpful is to synthetically generate variations of input images, such as by rotating them, or changing their brightness and contrast; this is called *data augmentation* and also works well for text and other types of model. We will be discussing it in detail in this chapter.\n", - "\n", - "Another point to consider is that although your problem might not look like a computer vision problem, it might be possible with a little imagination to turn it into one. For instance, if what you are trying to classify is sounds, you might try converting the sounds into images of their acoustic waveforms and then training a model on those images." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Text (natural language processing)**: just like in computer vision, computers are very good at categorising both short and long documents based on categories such as spam, sentiment, author, source website, and so forth. We are not aware of any rigorous work done in this area to compare to human performance, but anecdotally it seems to us that deep learning performance is similar to human performance here. Deep learning is also very good at generating context-appropriate text, such as generating replies to social media posts, and imitating a particular author's style. It is also good at making this content compelling to humans, and has been shown to be even more compelling than human-generated text. However, deep learning is currently not good at generating *correct* responses! We don't currently have a reliable way to, for instance, combine a knowledge base of medical information, along with a deep learning model for generating medically correct natural language responses. This is very dangerous, because it is so easy to create content which appears to a layman to be compelling, but actually is entirely incorrect.\n", - "\n", - "Another concern is that context-appropriate, highly compelling responses on social media can be used at massive scale — thousands of times greater than any troll farm previously seen — to spread disinformation, create unrest, and encourage conflict. As a rule of thumb, text generation will always be technologically a bit ahead of the ability of models to recognize automatically generated text. For instance, as we will see in this book, it is possible to use a model that can recognize artificially generated content to actually improve the generator that creates that content, until the classification model is no longer able to complete its task.\n", - "\n", - "Despite these issues, deep learning can be used to translate text from one language to another, summarize long documents into something which can be digested more quickly, find all mentions of a concept of interest, and so forth. Unfortunately, the translation or summary could well include completely incorrect information! However, it is already good enough that many people are using the systems — for instance Google's online translation system (and every other online service we are aware of) is based on deep learning." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Combining text and images**: the ability of deep learning to combine text and images into a single model is, generally, far better than most people intuitively expect. For example, a deep learning model can be trained on input images, and output captions written in English, and can learn to generate surprisingly appropriate captions automatically for new images! But again, we have the same warning that we discussed in the previous section: there is no guarantee that these captions will actually be correct.\n", - "\n", - "Because of this serious issue we generally recommend that deep learning be used not as a entirely automated process, but as part of a process in which the model and a human user interact closely. This can potentially make humans orders of magnitude more productive than they would be with entirely manual methods, and actually result in more accurate processes than using a human alone. For instance, an automatic system can be used to identify potential strokes directly from CT scans, send a high priority alert to have potential/scans looked at quickly. There is only a three-hour window to treat strokes, so this fast feedback loop could save lives. At the same time, however, all scans could continue to be sent to radiologists in the usual way, so there would be no reduction in human input. Other deep learning models could automatically measure items seen on the scan, and insert those measurements into report, warn the radiologist about findings that they may have missed, and tell the radiologist about other cases which might be relevant." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Tabular**: for analysing timeseries and tabular data, deep learning has recently been making great strides. However, deep learning is generally used as part of a ensemble of multiple types of model. If you already have a system that is using random forests or gradient boosting machines (popular tabular modelling tools that we will learn about soon) then switching to, or adding, deep learning may not result in any dramatic improvement. Deep learning does greatly increase the variety of columns that you can include, for example columns containing natural language (e.g. book titles, reviews, etc), and *high cardinality categorical* columns (i.e. something that contains a large number of discrete choices, such as zip code or product id). On the downside, deep learning models generally take longer to train than random forests or gradient boosting machines, although this is changing thanks to libraries such as [RAPIDS](https://rapids.ai/), which provides GPU acceleration for the whole modeling pipeline. We cover the pros and cons of all these methods in detail in <> in this book." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Recommendation systems**: Recommendation systems are really just a special type of tabular data. In particular, they generally have a high cardinality categorical variable representing users, and another one representing products (or something similar). A company like Amazon represents every purchase that has ever been made as a giant sparse matrix, with customers as the rows and products as the columns. Once they have the data in this format, data scientists apply some form of collaborative filtering to *fill in the matrix*. For example, if customer A buys products 1 and 10, and customer B buys products 1, 2, 4, and 10, the engine will recommend that A buy 2 and 4. Because deep learning models are good at handling high cardinality categorical variables they are quite good at handling recommendation systems. They particularly come into their own, just like for tabular data, when combining these variables with other kinds of data, such as natural language, or images. They can also do a good job of combining all of these types of information additional meta data represented as tables, such as user information, previous transactions, and so forth.\n", - "\n", - "However, nearly all machine learning approaches have the downside that the only tell you what products a particular user might like, rather than what recommendations would be helpful for a user. Many kinds of recommendations for products a user might like may not be at all helpful, for instance, if the user is already familiar with its products, or if they are simply different packagings of products they have already purchased (such as a boxed set of novels, where they already have each of the items in that set). Jeremy likes reading books by Terry Pratchett, and for a while Amazon was recommending nothing but Terry Pratchett books to him, which really wasn't helpful because he already was aware of these books!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\"Terry" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Other data types**: Often you will find that domain-specific data types fit very nicely into existing categories. For instance, protein chains look a lot like natural language documents, in that they are long sequences of discrete tokens with complex relationships and meaning throughout the sequence. And indeed, it does turn out the using NLP deep learning methods is the current state of the art approach for many types of protein analysis. As another example: sounds can be represented as spectrograms, which can be treated as images; standard deep learning approaches for images turn out to work really well on spectrograms." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### The Drivetrain approach" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "There are many accurate models that are of no use to anyone, and many inaccurate models that are highly useful. To ensure that your modeling work is useful in practice, you need to consider how your work will be used. In 2012 Jeremy, along with Margit Zwemer and Mike Loukides, introduced a method called *The Drivetrain Approach* for thinking about this issue, which we will summarize here. For more information, see the full article on oreilly.com [Designing Great Data Products](https://www.oreilly.com/radar/drivetrain-approach-data-products/).\n", - "\n", - "Consider a model in an autonomous vehicle, you want to help a car drive safely from point A to point B without human intervention. Great predictive modeling is an important part of the solution, but it doesn't stand on its own; as products become more sophisticated, it disappears into the plumbing. Someone using a self-driving car is completely unaware of the hundreds (if not thousands) of models and the petabytes of data that make it work. But as data scientists build increasingly sophisticated products, they need a systematic design approach.\n", - "\n", - "We use data not just to generate more data (in the form of predictions), but to produce *actionable outcomes*. That is the goal of the Drivetrain Approach. Start by defining a clear **objective**. For instance, Google, when creating their first search engine, considered \"What is the user’s main objective in typing in a search query?\", and their answer was \"show the most relevant search result\". The next step is to consider what **levers** you can pull (i.e. what actions could you take) to better achieve that objective. In Google's case, that was the ranking of the search results. The third step was to consider what new **data** they would need to produce such a ranking; they realized that the implicit information regarding which pages linked to which other pages could be used for this purpose. Only after these first three steps do we begin thinking about building the predictive **models**. Our objective and available levers, what data we already have and what additional data we will need to collect, determine the models we can build. The models will take both the levers and any uncontrollable variables as their inputs; the outputs from the models can be combined to predict the final state for our objective." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's consider another example: recommendation systems. The **objective** of a recommendation engine is to drive additional sales by surprising and delighting the customer with recommendations of items they would not have purchased without the recommendation. The **lever** is the ranking of the recommendations. New **data** must be collected to generate recommendations that will *cause new sales*. This will require conducting many randomized experiments in order to collect data about a wide range of recommendations for a wide range of customers. This is a step that few organizations take; but without it, you don't have the information you need to actually optimize recommendations based on your true objective (more sales!)\n", - "\n", - "Finally, you could build two **models** for purchase probabilities, conditional on seeing or not seeing a recommendation. The difference between these two probabilities is a utility function for a given recommendation to a customer. It will be low in cases where the algorithm recommends a familiar book that the customer has already rejected (both components are small) or a book that he or she would have bought even without the recommendation (both components are large and cancel each other out).\n", - "\n", - "As you can see, in practice often the practical implementation of your model will require a lot more than just training a model! You'll often need to run experiments to collect more data, and consider how to incorporate your models into the overall system you're developing." - ] - }, { "cell_type": "markdown", "metadata": {}, @@ -198,7 +89,160 @@ "\n", "Sometimes, you have to get a bit creative. Maybe you can find some previous machine learning project, such as a Kaggle competition, that is related to your field of interest. Sometimes, you have to compromize. Maybe you can't find the exact data you need for the precise project you have in mind; but you might be able to find something from a similar domain, or measured in a different way, tackling a slightly different problem. Working on these kinds of similar projects will still give you a good understanding of the overall process, and may help you identify other shortcuts, data sources, and so forth.\n", "\n", - "Especially when you are just starting out with deep learning it's not a good idea to branch out into very different areas to places that deep learning has not been applied to before. That's because if your model does not work at first, you will not know whether it is because you have made a mistake, or if the very problem you are trying to solve is simply not solvable with deep learning. And you won't know where to look to get help. Therefore, it is best at first to start with something where you can find an example online of somebody who has had good results with something that is at least somewhat similar to what you are trying to achieve, or where you can convert your data into a format similar what someone else has used before (such as creating an image from your data). Have a look at the *state of deep learning* earlier in this chapter for a reminder of what kinds of things deep learning is good at right now." + "Especially when you are just starting out with deep learning it's not a good idea to branch out into very different areas to places that deep learning has not been applied to before. That's because if your model does not work at first, you will not know whether it is because you have made a mistake, or if the very problem you are trying to solve is simply not solvable with deep learning. And you won't know where to look to get help. Therefore, it is best at first to start with something where you can find an example online of somebody who has had good results with something that is at least somewhat similar to what you are trying to achieve, or where you can convert your data into a format similar what someone else has used before (such as creating an image from your data). Let's have a look at the state of deep learning, jsut so you know what kinds of things deep learning is good at right now." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### The state of deep learning" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First things first, let's make sure that deep learning cn be any good at the problem you are considering. In general, here is a summary of the state of deep learning is at the start of 2020. However, things move very fast, and by the time you read this some of these constraints may no longer exist. We will try to keep the book website up-to-date; in addition, a Google search for \"what can AI do now\" there is likely to provide some up-to-date information." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Computer vision" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There are many domains in which deep learning has not been used to analyse images yet, but those where it has been tried have nearly universally shown that computers can recognise what items are in an image at least as well as people can — even specially trained people, such as radiologists. This is known as *object recognition*. Deep learning is also good at recognizing whereabouts objects in an image are, and can highlight their location and name each found object. This is known as *object detection* (there is also a variant of this we saw in <>, where every pixel is categorized based on what kind of object it is part of--this is called *segmentation*). Deep learning algorithms are generally not good at recognizing images that are significantly different in structure or style to those used to train the model. For instance, if there were no black-and-white images in the training data, the model may well do poorly on black-and-white images. If the training data did not contain hand-drawn images then the model will probably do poorly on hand-drawn images. There is no general way to check what types of image are missing in your training set, but we will show in this chapter some ways to try to recognize when unexpected image types arise in the data when the model is being used in production (this is known as checking for *out of domain* data).\n", + "\n", + "One major challenge for object detection systems is that image labelling can be slow and expensive. There is a lot of work at the moment going into tools to try to make this labelling faster and more easy, and require less handcrafted labels to train accurate object detection models. One approach which is particularly helpful is to synthetically generate variations of input images, such as by rotating them, or changing their brightness and contrast; this is called *data augmentation* and also works well for text and other types of model. We will be discussing it in detail in this chapter.\n", + "\n", + "Another point to consider is that although your problem might not look like a computer vision problem, it might be possible with a little imagination to turn it into one. For instance, if what you are trying to classify is sounds, you might try converting the sounds into images of their acoustic waveforms and then training a model on those images." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Text (natural language processing)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Just like in computer vision, computers are very good at categorising both short and long documents based on categories such as spam, sentiment, author, source website, and so forth. We are not aware of any rigourous work done in this area to compare to human performance, but anecdotally it seems to us that deep learning performance is similar to human performance here. Deep learning is also very good at generating context-appropriate text, such as generating replies to social media posts, and imitating a particular author's style. It is also good at making this content compelling to humans, and has been shown to be even more compelling than human-generated text. However, deep learning is currently not good at generating *correct* responses! We don't currently have a reliable way to, for instance, combine a knowledge base of medical information, along with a deep learning model for generating medically correct natural language responses. This is very dangerous, because it is so easy to create content which appears to a layman to be compelling, but actually is entirely incorrect.\n", + "\n", + "Another concern is that context-appropriate, highly compelling responses on social media can be used at massive scale — thousands of times greater than any troll farm previously seen — to spread disinformation, create unrest, and encourage conflict. As a rule of thumb, text generation will always be technologically a bit ahead of the ability of models to recognize automatically generated text. For instance, it is possible to use a model that can recognize artificially generated content to actually improve the generator that creates that content, until the classification model is no longer able to complete its task.\n", + "\n", + "Despite these issues, deep learning can be used to translate text from one language to another, summarize long documents into something which can be digested more quickly, find all mentions of a concept of interest, and so forth. Unfortunately, the translation or summary could well include completely incorrect information! However, it is already good enough that many people are using the systems — for instance Google's online translation system (and every other online service we are aware of) is based on deep learning." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Combining text and images" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The ability of deep learning to combine text and images into a single model is, generally, far better than most people intuitively expect. For example, a deep learning model can be trained on input images, and output captions written in English, and can learn to generate surprisingly appropriate captions automatically for new images! But again, we have the same warning that we discussed in the previous section: there is no guarantee that these captions will actually be correct.\n", + "\n", + "Because of this serious issue we generally recommend that deep learning be used not as a entirely automated process, but as part of a process in which the model and a human user interact closely. This can potentially make humans orders of magnitude more productive than they would be with entirely manual methods, and actually result in more accurate processes than using a human alone. For instance, an automatic system can be used to identify potential strokes directly from CT scans, send a high priority alert to have potential/scans looked at quickly. There is only a three-hour window to treat strokes, so this fast feedback loop could save lives. At the same time, however, all scans could continue to be sent to radiologists in the usual way, so there would be no reduction in human input. Other deep learning models could automatically measure items seen on the scan, and insert those measurements into report, warn the radiologist about findings that they may have missed, and tell the radiologist about other cases which might be relevant." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Tabular data" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For analysing timeseries and tabular data, deep learning has recently been making great strides. However, deep learning is generally used as part of a ensemble of multiple types of model. If you already have a system that is using random forests or gradient boosting machines (popular tabular modelling tools that we will learn about soon) then switching to, or adding, deep learning may not result in any dramatic improvement. Deep learning does greatly increase the variety of columns that you can include, for example columns containing natural language (e.g. book titles, reviews, etc), and *high cardinality categorical* columns (i.e. something that contains a large number of discrete choices, such as zip code or product id). On the downside, deep learning models generally take longer to train than random forests or gradient boosting machines, although this is changing thanks to libraries such as [RAPIDS](https://rapids.ai/), which provides GPU acceleration for the whole modeling pipeline. We cover the pros and cons of all these methods in detail in <> in this book." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Recommendation systems" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Recommendation systems are really just a special type of tabular data. In particular, they generally have a high cardinality categorical variable representing users, and another one representing products (or something similar). A company like Amazon represents every purchase that has ever been made as a giant sparse matrix, with customers as the rows and products as the columns. Once they have the data in this format, data scientists apply some form of collaborative filtering to *fill in the matrix*. For example, if customer A buys products 1 and 10, and customer B buys products 1, 2, 4, and 10, the engine will recommend that A buy 2 and 4. Because deep learning models are good at handling high cardinality categorical variables they are quite good at handling recommendation systems. They particularly come into their own, just like for tabular data, when combining these variables with other kinds of data, such as natural language, or images. They can also do a good job of combining all of these types of information with additional meta data represented as tables, such as user information, previous transactions, and so forth.\n", + "\n", + "However, nearly all machine learning approaches have the downside that they only tell you what products a particular user might like, rather than what recommendations would be helpful for a user. Many kinds of recommendations for products a user might like may not be at all helpful, for instance, if the user is already familiar with its products, or if they are simply different packagings of products they have already purchased (such as a boxed set of novels, where they already have each of the items in that set). Jeremy likes reading books by Terry Pratchett, and for a while Amazon was recommending nothing but Terry Pratchett books to him (see <>), which really wasn't helpful because he already was aware of these books!" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\"Terry" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Other data types**: Often you will find that domain-specific data types fit very nicely into existing categories. For instance, protein chains look a lot like natural language documents, in that they are long sequences of discrete tokens with complex relationships and meaning throughout the sequence. And indeed, it does turn out that using NLP deep learning methods is the current state of the art approach for many types of protein analysis. As another example: sounds can be represented as spectrograms, which can be treated as images; standard deep learning approaches for images turn out to work really well on spectrograms." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "TK Jeremy: please add a transition" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### The Drivetrain approach" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There are many accurate models that are of no use to anyone, and many inaccurate models that are highly useful. To ensure that your modeling work is useful in practice, you need to consider how your work will be used. In 2012 Jeremy, along with Margit Zwemer and Mike Loukides, introduced a method called *The Drivetrain Approach* for thinking about this issue, which we will summarize here, and illustrate in <>. For more information, see the full article on oreilly.com [Designing Great Data Products](https://www.oreilly.com/radar/drivetrain-approach-data-products/).\n", + "\n", + "Consider a model in an autonomous vehicle, you want to help a car drive safely from point A to point B without human intervention. Great predictive modeling is an important part of the solution, but it doesn't stand on its own; as products become more sophisticated, it disappears into the plumbing. Someone using a self-driving car is completely unaware of the hundreds (if not thousands) of models and the petabytes of data that make it work. But as data scientists build increasingly sophisticated products, they need a systematic design approach.\n", + "\n", + "We use data not just to generate more data (in the form of predictions), but to produce *actionable outcomes*. That is the goal of the Drivetrain Approach. Start by defining a clear **objective**. For instance, Google, when creating their first search engine, considered \"What is the user’s main objective in typing in a search query?\", and their answer was \"show the most relevant search result\". The next step is to consider what **levers** you can pull (i.e. what actions could you take) to better achieve that objective. In Google's case, that was the ranking of the search results. The third step was to consider what new **data** they would need to produce such a ranking; they realized that the implicit information regarding which pages linked to which other pages could be used for this purpose. Only after these first three steps do we begin thinking about building the predictive **models**. Our objective and available levers, what data we already have and what additional data we will need to collect, determine the models we can build. The models will take both the levers and any uncontrollable variables as their inputs; the outputs from the models can be combined to predict the final state for our objective." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's consider another example: recommendation systems. The **objective** of a recommendation engine is to drive additional sales by surprising and delighting the customer with recommendations of items they would not have purchased without the recommendation. The **lever** is the ranking of the recommendations. New **data** must be collected to generate recommendations that will *cause new sales*. This will require conducting many randomized experiments in order to collect data about a wide range of recommendations for a wide range of customers. This is a step that few organizations take; but without it, you don't have the information you need to actually optimize recommendations based on your true objective (more sales!)\n", + "\n", + "Finally, you could build two **models** for purchase probabilities, conditional on seeing or not seeing a recommendation. The difference between these two probabilities is a utility function for a given recommendation to a customer. It will be low in cases where the algorithm recommends a familiar book that the customer has already rejected (both components are small) or a book that he or she would have bought even without the recommendation (both components are large and cancel each other out).\n", + "\n", + "As you can see, in practice often the practical implementation of your model will require a lot more than just training a model! You'll often need to run experiments to collect more data, and consider how to incorporate your models into the overall system you're developing. Speaking of data, let's now focus on how to find find data for your project." ] }, { @@ -483,7 +527,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "One thing to be aware of in this process: as we discussed in <>, models can only reflect the data used to train them. And the world is full of biased data, which ends up reflected in, for example, Bing Image Search (which we used to create our dataset). For instance, let's say you were interested in creating an app which could help users figure out whether they had healthy skin, so you trained a model on the results of searches for (say) *healthy skin*. Here's the results you would get:" + "One thing to be aware of in this process: as we discussed in <>, models can only reflect the data used to train them. And the world is full of biased data, which ends up reflected in, for example, Bing Image Search (which we used to create our dataset). For instance, let's say you were interested in creating an app which could help users figure out whether they had healthy skin, so you trained a model on the results of searches for (say) *healthy skin*. <> shows you the results you would get." ] }, { @@ -500,6 +544,13 @@ "So with this as your training data, you would end up not with a healthy skin detector, but a *young white woman touching her face* detector! Be sure to think carefully about the types of data that you might expect to see in practice in your application, and check carefully to ensure that all these types are reflected in your model's source data. (Thanks to Deb Raji, who came up with the *healthy skin* example. See her paper *Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products* for more fascinating insights into model bias.)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now that we have downloaded some data, we need to assemble it in a format suitable for model training. In fastai, that means creating an object called `DataLoaders`." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -697,14 +748,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Data augmentation" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "All of these approaches seem somewhat wasteful, or problematic. If we squished or stretch the images then the end up unrealistic shapes, leading to a model that learns that things look different to how they actually are, which we would expect to result in lower accuracy. If we crop the images then we remove some of the features that allow us to recognize them. For instance, if we were trying to recognise the breed of dog or cat, we may end up cropping out a key part of the body or the face necessary to distinguish between similar breeds. If we pat the images then we have a whole lot of empty space, which is just wasted computation for our model, and results in a lower effective resolution for the part of the image we actually use.\n", + "All of these approaches seem somewhat wasteful, or problematic. If we squished or stretch the images then the end up unrealistic shapes, leading to a model that learns that things look different to how they actually are, which we would expect to result in lower accuracy. If we crop the images then we remove some of the features that allow us to recognize them. For instance, if we were trying to recognise the breed of dog or cat, we may end up cropping out a key part of the body or the face necessary to distinguish between similar breeds. If we pad the images then we have a whole lot of empty space, which is just wasted computation for our model, and results in a lower effective resolution for the part of the image we actually use.\n", "\n", "Instead, what we normally do in practice is to randomly select part of the image, and crop to just that part. On each epoch (which is one complete pass through all of our images in the dataset) we randomly select a different part of each image. This means that our model can learn to focus on, and recognize, different features in our images. It also reflects how images work in the real world; different photos of the same thing may be framed in slightly different ways.\n", "\n", @@ -749,7 +793,21 @@ "source": [ "In fact, an entirely untrained neural network knows nothing whatsoever about how images behave. It doesn't even recognise that when an object is moved one pixel to the left, then it still is a picture of the same thing! So actually training the neural network with examples of images that are in slightly different places, and slightly different sizes, helps it to understand the basic concept of what a *object* is, and how it can be represented in an image.\n", "\n", - "This is a specific example of a more general technique, called *data augmentation*. Data augmentation refers to creating random variations of our input data, such that they appear a different, but are not expected to change the meaning of the data. Examples of common data augmentation for images are rotation, flipping, perspective warping, brightness changes, contrast changes, and much more. For natural photo images such as the ones we are using here, there is a standard set of augmentations which we have found work pretty well, and are provided with the get transforms function. Because the images are now all the same size, we can apply these augmentation is to an entire batch of the time using the GPU, which will save a lot of time. To tell fastai we want to use these transforms to a batch, we use the `batch_tfms` parameter. (Note that's we're not using `RandomResizedCrop` in this example, so you can see the differences more clearly; we're also using double the amount of augmentation compared to the default, for the same reason)." + "This is a specific example of a more general technique, called *data augmentation*." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Data augmentation" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Data augmentation refers to creating random variations of our input data, such that they appear a different, but are not expected to change the meaning of the data. Examples of common data augmentation for images are rotation, flipping, perspective warping, brightness changes, contrast changes, and much more. For natural photo images such as the ones we are using here, there is a standard set of augmentations which we have found work pretty well, and are provided with the get transforms function. Because the images are now all the same size, we can apply these augmentations to an entire batch of them using the GPU, which will save a lot of time. To tell fastai we want to use these transforms to a batch, we use the `batch_tfms` parameter. (Note that's we're not using `RandomResizedCrop` in this example, so you can see the differences more clearly; we're also using double the amount of augmentation compared to the default, for the same reason)." ] }, { @@ -777,6 +835,13 @@ "dls.train.show_batch(max_n=8, rows=2)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now that we have assembled our data in a format fit for model training, let's actually train an image classifier using it." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -788,7 +853,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We'll use `RandomResizedCrop` and default `aug_transforms` for our model, and an image size of 224px, which is fairly standard for image classification." + "Time to use the same lined of codes as in <> to train our bear classifier.\n", + "\n", + "We don't have a lot of data for our pblem (150 pictures of each sort of bear at most), so to train our model, we'll use `RandomResizedCrop` and default `aug_transforms` for our model, on an image size of 224px, which is fairly standard for image classification." ] }, { @@ -1067,6 +1134,13 @@ "> note: After cleaning the dataset using the above steps, we generally are seeing 100% accuracy on this task. We even see that result when we download a lot less images than the 150 per class we're using here. As you can see, the common complaint *you need massive amounts of data to do deep learning* can be a very long way from the truth!" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now that we have trained our model, let's see how we can deploy it to be used in practice." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1074,6 +1148,13 @@ "## Turning your model into an online application" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We are now going to look at what it takes to take this model and turn it into a working online application. We will just go as far as creating a basic working prototype; we do not have the scope in this book to teach you all the details of web application development generally." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1085,8 +1166,6 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We are now going to look at what it takes to take this model and turn it into a working online application. We will just go as far as creating a basic working prototype; we do not have the scope in this book to teach you all the details of web application development generally.\n", - "\n", "Once you've got a model you're happy with, you need to save it, so that you can then copy it over to a server where you'll use it in production. Do you remember exactly what a model is? It consists of two parts: the *architecture*, and the trained *parameters*. The easiest way to save a model is to save both of these, because that way when you load a model you can be sure that you have the matching architecture and parameters. To save both parts, use the `export` method.\n", "\n", "This method even saves the definition of how to create your `DataLoaders`. This is important, because otherwise you would have to redefine how to transform your data in order to use your model in production. When you call export, fastai will save a file called `export.pkl`." @@ -1218,6 +1297,13 @@ "We can see here that if we index into the vocab with the integer returned by `predict` then we get back \"grizzly\", as expected. Also, note that if we index into the list of probabilities, we see a nearly 1.00 probability that this is a grizzly." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We know how to make predictions from our saved model, so we have everything we need to start building our app. We can do it directly in a Jupyter Notenook." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1500,6 +1586,13 @@ "\"The" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We have written all the code necessary for our app. The next step is to convert it in something we can deploy." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1522,7 +1615,9 @@ "\n", "Voila runs Jupyter notebooks, just like the Jupyter notebook server you are using now does, except that it does something very important: it removes all of the cell inputs, and only shows output (including ipywidgets), along with your markdown cells. So what's left is a web application! To view your notebook as a voila web application replace the word \"notebooks\" in your browser's URL with: \"voila/render\". You will see the same content as your notebook, but without any of the code cells.\n", "\n", - "Of course, you don't need to use Voila or ipywidgets. Your model is just a function you can call: `pred,pred_idx,probs = learn.predict(img)` . So you can use it with any framework, hosted on any platform. And you can take something you've prototyped in ipywidgets and Voila and later convert it into a regular web application. We're showing you this approach in the book because we think it's a great way for data scientists and other folks that aren't web development experts to create applications from their models." + "Of course, you don't need to use Voila or ipywidgets. Your model is just a function you can call: `pred,pred_idx,probs = learn.predict(img)` . So you can use it with any framework, hosted on any platform. And you can take something you've prototyped in ipywidgets and Voila and later convert it into a regular web application. We're showing you this approach in the book because we think it's a great way for data scientists and other folks that aren't web development experts to create applications from their models.\n", + "\n", + "We have our app, now let's deploy it!" ] }, { @@ -1553,7 +1648,7 @@ "For at least the initial prototype of your application, and for any hobby projects that you want to show off, you can easily host them for free. The best place and the best way to do this will vary over time so check the book website for the most up-to-date recommendations. As we're writing this book in 2020 the simplest (and free!) approach is called [Binder](https://mybinder.org/). To publish your web app on Binder, you follow these steps:\n", "\n", "1. Add your notebook to a [GitHub repository](http://github.com/), \n", - "2. Paste the URL of that repo in the URL field of Binder, \n", + "2. Paste the URL of that repo in the URL field of Binder as shown in <>, \n", "3. Change the \"File\" dropdown to instead select \"URL\",\n", "4. In the Path field, enter `/voila/render/name.ipynb` (replacing `name.ipynb` as appropriate for your notebook):\n", "5. Click the \"Copy the URL\" button and paste it somewhere safe. \n", @@ -1595,7 +1690,9 @@ "source": [ "> A: I've had a chance to see up close how the mobile ML landscape is changing in my work. We offer an iPhone app that depends on computer vision and for years we ran our own computer vision models in the cloud. This was the only way to do it then since those models needed significant memory and compute resources and took minutes to process. This approach required building not only the models (fun!) but infrastructure to ensure a certain number of \"compute worker machines\" was absolutely always running (scary), that more machines would automatically come online if traffic increased, that there was stable storage for large inputs and outputs, that the iOS app could know and tell the user how their job was doing, etc... Nowadays, Apple provides APIs for converting models to run efficiently on device and most iOS devices have dedicated ML hardware, so we run our new models on device. So, in a few years that strategy has gone from impossible to possible but it's still not easy. In our case it's worth it, for a faster user experiene and to worry less about servers. What works for you will depend, realistically, on the user experience you're trying to create and what you personally find it easy to do. If you really know how to run servers, do it. If you really know how to build native mobile apps, do that. There are many roads up the hill.\n", "\n", - "Overall, we'd recommend using a simple CPU-based server approach where possible, for as long as you can get away with it. If you're lucky enough to have a very successful application, then you'll be able to justify the investment in more complex deployment approaches at that time." + "Overall, we'd recommend using a simple CPU-based server approach where possible, for as long as you can get away with it. If you're lucky enough to have a very successful application, then you'll be able to justify the investment in more complex deployment approaches at that time.\n", + "\n", + "Congratulations, you have succesfully built a deep learning model and deployed it! Now is a good time to take a pause and think about what could go wrong." ] }, { @@ -1609,7 +1706,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "In practice, a deep learning model will be just one piece of a much bigger system. As we discussed at the start of this chapter, a *data product* requires thinking about the entire end to end process within which or model lives.\n", + "In practice, a deep learning model will be just one piece of a much bigger system. As we discussed at the start of this chapter, a *data product* requires thinking about the entire end to end process within which our model lives.\n", "\n", "One of the biggest issues with this is that understanding and testing the behavior of a deep learning model is much more difficult than most code that you would write. With normal software development you can analyse the exact steps that the software is taking, and carefully study with of these steps match the desired behaviour that you are trying to create. But with a neural network the behavior emerges from the models attempt to match the training data, rather than being exactly defined.\n", "\n", @@ -1632,7 +1729,7 @@ "\n", "There are other reasons we need to be careful too. One very common problem is *domain shift*; this is where the type of data that our model sees changes over time. For instance, an insurance company may use a deep learning model as part of their pricing and risk algorithm, but over time the type of customers that they attract, and the type of risks that they represent, may change so much that the original training data is no longer relevant.\n", "\n", - "Out of domain data, and domain shift, are examples of the problem that you can never fully no the entire behaviour of your neural network. They have far too many parameters to be able to analytically understand all of their possible behaviours. This is the natural downside of the thing that they're so good at — their flexibility in being able to solve complex problems where we may not even be able to fully specify our preferred solution approaches. The good news, however, is that there are ways to mitigate these risks using a carefully thought out process. The details of this will vary depending on the details of the problem you are solving, but we will attempt to lay out here a high-level approach which we hope will provide useful guidance." + "Out of domain data, and domain shift, are examples of the problem that you can never fully know the entire behaviour of your neural network. They have far too many parameters to be able to analytically understand all of their possible behaviours. This is the natural downside of the thing that they're so good at — their flexibility in being able to solve complex problems where we may not even be able to fully specify our preferred solution approaches. The good news, however, is that there are ways to mitigate these risks using a carefully thought out process. The details of this will vary depending on the details of the problem you are solving, but we will attempt to lay out here a high-level approach summarized in <> which we hope will provide useful guidance." ] }, { @@ -1660,6 +1757,13 @@ "> j: I started a company 20 years ago called *Optimal Decisions* which used machine learning and optimisation to help giant insurance companies set their pricing, impacting tens of billions of dollars of risks. We used the approaches described above to manage the potential downsides of something that might go wrong. Also, before we worked with our clients to put anything in production, we tried to simulate the impact by testing the end to end system on their previous year's data. It was always quite a nerve-wracking process, putting these new algorithms in production, but every rollout was successful." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As you analyze the results while deploying your model progressively, you should check for the following unexpected behaviors." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1680,6 +1784,13 @@ "Such a thought exercise might help you to construct a more careful rollout plan, ongoing monitoring systems, and human oversight. Of course, human oversight isn't useful if it isn't listened to; so make sure that there are reliable and resilient communication channels so that the right people will be aware of issues, and will have the power to fix them." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Congratulations, you have finished your first deep learning project! To help with understanding the material, we really recommend you start writing about what you learned." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1785,6 +1896,31 @@ "display_name": "Python 3", "language": "python", "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.5" + }, + "toc": { + "base_numbering": 1, + "nav_menu": {}, + "number_sections": false, + "sideBar": true, + "skip_h1_title": true, + "title_cell": "Table of Contents", + "title_sidebar": "Contents", + "toc_cell": false, + "toc_position": {}, + "toc_section_display": true, + "toc_window_display": false } }, "nbformat": 4,