added README.rst files from Atlas
This commit is contained in:
parent
5774e57b60
commit
d9e8f44297
4
01-data-model/README.rst
Normal file
4
01-data-model/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 1 - "The Python Data Model"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
02-array-seq/README.rst
Normal file
4
02-array-seq/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 2 - "An array of sequences"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
03-dict-set/README.rst
Normal file
4
03-dict-set/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 3 - "Dictionaries and sets"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
04-text-byte/README.rst
Normal file
4
04-text-byte/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 4 - "Text and bytes"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
05-1class-func/README.rst
Normal file
4
05-1class-func/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 5 - "First-class functions"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
06-dp-1class-func/README.rst
Normal file
4
06-dp-1class-func/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 6 - "Design patterns with first class functions"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
07-closure-deco/README.rst
Normal file
4
07-closure-deco/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 7 - "Closures and decorators"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
08-obj-ref/README.rst
Normal file
4
08-obj-ref/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 8 - "Object references, mutability and recycling"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
09-pythonic-obj/README.rst
Normal file
4
09-pythonic-obj/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 9 - "Pythonic objects"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
10-seq-hacking/README.rst
Normal file
4
10-seq-hacking/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 10 - "Sequence hacking, hashing and slicing"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
11-iface-abc/README.rst
Normal file
4
11-iface-abc/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 11 - "Interfaces, protocols and ABCs"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
12-inheritance/README.rst
Normal file
4
12-inheritance/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 12 - "Inheritance: for good or for worse"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
13-op-overloading/README.rst
Normal file
4
13-op-overloading/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 13 - "Operator overloading: doing it right"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
14-it-generator/README.rst
Normal file
4
14-it-generator/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 14 - "Iterables, iterators and generators"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
15-context-mngr/README.rst
Normal file
4
15-context-mngr/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 15 - "Context managers and something else"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
16-coroutine/README.rst
Normal file
4
16-coroutine/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 16 - "Coroutines"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
17-futures/README.rst
Normal file
4
17-futures/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 17 - "Concurrency with futures"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
194
17-futures/countries/README.rst
Normal file
194
17-futures/countries/README.rst
Normal file
@ -0,0 +1,194 @@
|
||||
=====================================================
|
||||
Setting up Nginx and Vaurien for testing HTTP clients
|
||||
=====================================================
|
||||
|
||||
This text explains how to configure Nginx and Vaurien to test HTTP client code while avoiding network traffic and introducing simulated delays and errors. This setup is necessary if you want to experiment with the image download examples in chapters 17 and 18 of Fluent Python.
|
||||
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
The flag download examples are designed to compare the performance of different approaches to finding and downloading files from the Web. However, we don't want to hit a public server with multiple requests per second while testing, and we want to be able to simulate high latency and random network errors.
|
||||
|
||||
For this setup I chose Nginx as the HTTP server because it is very fast and easy to configure, and the Vaurien proxy because it was designed by Mozilla to introduce delays and network errors for testing Web services.
|
||||
|
||||
The archive ``countries/flags.zip``, contains a directory ``flags/`` with 194 subdirectories, each containing a ``.gif`` image and a ``metadata.json`` file. These are public-domain images copied from the `CIA World Fact Book <https://www.cia.gov/library/publications/the-world-factbook/>`_.
|
||||
|
||||
Once these files are unpacked to the ``flags/`` directory and Nginx is configured, you can experiment with the ``flags*.py``examples without hitting the network.
|
||||
|
||||
|
||||
Procedure
|
||||
=========
|
||||
|
||||
1. Unpack test data
|
||||
-------------------
|
||||
|
||||
The instructions in this section are for GNU/Linux or OSX using the command line. Windows users should have no difficulty doing the same operations with the Windows Exporer GUI.
|
||||
|
||||
Unpack the initial data in the ``countries/`` directory::
|
||||
|
||||
$ unzip flags.zip
|
||||
... many lines omitted ...
|
||||
creating: flags/zw/
|
||||
inflating: flags/zw/metadata.json
|
||||
inflating: flags/zw/zw.gif
|
||||
|
||||
|
||||
Verify that 194 directories are created in ``countries/flags/``, each with a ``.gif`` and a ``metadata.json`` file::
|
||||
|
||||
|
||||
$ ls flags | wc -w
|
||||
194
|
||||
$ find flags | grep .gif | wc -l
|
||||
194
|
||||
$ find flags | grep .json | wc -l
|
||||
194
|
||||
$ ls flags/ad
|
||||
ad.gif metadata.json
|
||||
|
||||
|
||||
2. Install Nginx
|
||||
----------------
|
||||
|
||||
Download and install Nginx. I used version 1.6.2 -- the latest stable version as I write this.
|
||||
|
||||
* Download page: http://nginx.org/en/download.html
|
||||
|
||||
* Beginner's guide: http://nginx.org/en/docs/beginners_guide.html
|
||||
|
||||
|
||||
3. Configure Nginx
|
||||
------------------
|
||||
|
||||
Edit the the ``nginx.conf`` file to set the port and document root. You can determine which ``nginx.conf`` is in use by running::
|
||||
|
||||
|
||||
$ nginx -V
|
||||
|
||||
|
||||
The output starts with::
|
||||
|
||||
nginx version: nginx/1.6.2
|
||||
built by clang 6.0 (clang-600.0.51) (based on LLVM 3.5svn)
|
||||
TLS SNI support enabled
|
||||
configure arguments:...
|
||||
|
||||
|
||||
Among the configure arguments you'll see ``--conf-path=``. That's the file you will edit.
|
||||
|
||||
Most of the content in ``nginx.conf`` is within a block labeled ``http`` and enclosed in curly braces. Within that block there can be multiple blocks labeled ``server``. Add another ``server`` block like this one::
|
||||
|
||||
|
||||
server {
|
||||
listen 8001;
|
||||
|
||||
location /flags/ {
|
||||
root /full-path-to.../countries/;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
After editing ``nginx.conf`` the server must be started (if it's not running) or told to reload the configuration file:
|
||||
|
||||
|
||||
$ nginx # to start, if necessary
|
||||
$ nginx -s reload # to reload the configuration
|
||||
|
||||
|
||||
To test the configuration, open the URL http://localhost:8001/flags/ad/ad.gif in a browser. You should see the blue, yellow and red flag of Andorra.
|
||||
|
||||
If the test fails, please double check the procedure just described and refer to the Nginx documentation.
|
||||
|
||||
At this point you may run the ``flags_*2.py`` examples against the Nginx install by providing the ``--server LOCAL`` command line option. For example::
|
||||
|
||||
|
||||
$ python3 flags2_threadpool.py -s LOCAL
|
||||
LOCAL site: http://localhost:8001/flags
|
||||
Searching for 20 flags: from BD to VN
|
||||
20 concurrent connections will be used.
|
||||
--------------------
|
||||
20 flags downloaded.
|
||||
Elapsed time: 0.09s
|
||||
|
||||
|
||||
Note that Nginx is so fast that you will not see much difference in run time between the sequential and the concurrent versions. For more realistic testing with simulated network lag, we need to set up the Vaurien proxy.
|
||||
|
||||
|
||||
4. Install and run Vaurien
|
||||
--------------------------
|
||||
|
||||
Vaurien depends on gevent which is only available for Python 2.5-2.7. To install Vaurien I opened another shell, created a virtualenv for Python 2.7, and used that environment to install and run the ``vaurien`` package::
|
||||
|
||||
|
||||
$ virtualenv-2.7 .env27 --no-site-packages --distribute
|
||||
New python executable in .env27/bin/python
|
||||
Installing setuptools, pip...done.
|
||||
$ . .env27/bin/activate
|
||||
(.env27)$ pip install vaurien
|
||||
Downloading/unpacking vaurien
|
||||
Downloading vaurien-1.9.tar.gz (50kB): 50kB downloaded
|
||||
...many lines and a few minutes later...
|
||||
|
||||
Successfully installed vaurien cornice gevent statsd-client vaurienclient
|
||||
greenlet http-parser pyramid simplejson requests zope.interface
|
||||
translationstring PasteDeploy WebOb repoze.lru zope.deprecation venusian
|
||||
Cleaning up...
|
||||
|
||||
|
||||
Using that same shell with the ``.env27`` environment active, run the ``vaurien_delay.sh`` script in the ``countries/`` directory::
|
||||
|
||||
|
||||
(.env27)$ $ ./vaurien_delay.sh
|
||||
2015-02-25 20:20:17 [69124] [INFO] Starting the Chaos TCP Server
|
||||
2015-02-25 20:20:17 [69124] [INFO] Options:
|
||||
2015-02-25 20:20:17 [69124] [INFO] * proxies from localhost:8002 to localhost:8001
|
||||
2015-02-25 20:20:17 [69124] [INFO] * timeout: 30
|
||||
2015-02-25 20:20:17 [69124] [INFO] * stay_connected: 0
|
||||
2015-02-25 20:20:17 [69124] [INFO] * pool_max_size: 100
|
||||
2015-02-25 20:20:17 [69124] [INFO] * pool_timeout: 30
|
||||
2015-02-25 20:20:17 [69124] [INFO] * async_mode: 1
|
||||
|
||||
|
||||
The ``vaurien_delay.sh`` creates an HTTP proxy on port 8002 which adds a 1s delay to every response. You can test it with a browser on port 8002: http://localhost:8002/flags/ad/ad.gif -- the flag of Andorra should appear after 1 second.
|
||||
|
||||
There is also the ``vaurien_error_delay.sh`` script which runs a proxy on port 8003 producing errors in 25% of the responses and a .5 se delay to 50% of the responses. You can also test it with the browser on port 8003, but rememeber that errors are expected.
|
||||
|
||||
|
||||
Platform-specific instructions
|
||||
==============================
|
||||
|
||||
|
||||
Nginx setup on Mac OS X
|
||||
------------------------
|
||||
|
||||
Homebrew (copy & paste code at the bottom of http://brew.sh/):
|
||||
|
||||
|
||||
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
|
||||
$ brew doctor
|
||||
$ brew install nginx
|
||||
|
||||
|
||||
Download and unpack::
|
||||
|
||||
Docroot is: /usr/local/var/www
|
||||
/usr/local/etc/nginx/nginx.conf
|
||||
|
||||
|
||||
::
|
||||
|
||||
To have launchd start nginx at login:
|
||||
ln -sfv /usr/local/opt/nginx/*.plist ~/Library/LaunchAgents
|
||||
Then to load nginx now:
|
||||
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.nginx.plist
|
||||
Or, if you don't want/need launchctl, you can just run:
|
||||
nginx
|
||||
|
||||
|
||||
|
||||
Nginx setup on Lubuntu 14.04.1 LTS
|
||||
----------------------------------
|
||||
|
||||
Docroot is: /usr/share/nginx/html
|
||||
|
||||
|
4
18-asyncio/README.rst
Normal file
4
18-asyncio/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 18 - "Concurrency with asyncio"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
3
18-asyncio/countries/README.rst
Normal file
3
18-asyncio/countries/README.rst
Normal file
@ -0,0 +1,3 @@
|
||||
The ``asyncio`` flag download examples are in the
|
||||
``../../17-futures/countries/`` directory together
|
||||
with the sequential and threadpool examples.
|
4
19-dyn-attr-prop/README.rst
Normal file
4
19-dyn-attr-prop/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 19 - "Dynamic attributes and properties"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
20-descriptor/README.rst
Normal file
4
20-descriptor/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 20 - "Attribute descriptors"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
4
21-class-metaprog/README.rst
Normal file
4
21-class-metaprog/README.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Sample code for Chapter 21 - "Class metaprogramming"
|
||||
|
||||
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
|
||||
http://shop.oreilly.com/product/0636920032519.do
|
33
concurrency/flags/README.rst
Normal file
33
concurrency/flags/README.rst
Normal file
@ -0,0 +1,33 @@
|
||||
=========================================
|
||||
Setting up the test environment
|
||||
=========================================
|
||||
|
||||
Some of the concurrency examples in this book require a local HTTP
|
||||
server. These instructions show how I setup Ngnix on GNU/Linux,
|
||||
Mac OS X 10.9 and Windows 7.
|
||||
|
||||
Nginx setup on Mac OS X
|
||||
========================
|
||||
|
||||
Homebrew (copy & paste code at the bottom of http://brew.sh/)::
|
||||
|
||||
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
|
||||
$ brew doctor
|
||||
$ brew install nginx
|
||||
|
||||
Download and unpack::
|
||||
|
||||
Docroot is: /usr/local/var/www
|
||||
/usr/local/etc/nginx/nginx.conf
|
||||
|
||||
To have launchd start nginx at login:
|
||||
ln -sfv /usr/local/opt/nginx/*.plist ~/Library/LaunchAgents
|
||||
Then to load nginx now:
|
||||
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.nginx.plist
|
||||
Or, if you don't want/need launchctl, you can just run:
|
||||
nginx
|
||||
|
||||
Nginx setup on Lubuntu 14.04.1 LTS
|
||||
==================================
|
||||
|
||||
/usr/share/nginx/html
|
138
concurrency/wikipedia/README.rst
Normal file
138
concurrency/wikipedia/README.rst
Normal file
@ -0,0 +1,138 @@
|
||||
====================================
|
||||
Configuring a local test environment
|
||||
====================================
|
||||
|
||||
tl;dr;
|
||||
======
|
||||
|
||||
This text explains how to configure **nginx** and **vaurien** to build
|
||||
a local mirror of the data to run the Wikipedia Picture of the Day
|
||||
examples while avoiding network traffic and introducing controlled
|
||||
delays and errors for testing, thanks to the **vaurien** proxy.
|
||||
|
||||
|
||||
Rationale and overview
|
||||
======================
|
||||
|
||||
The Wikipedia Picture of the Day examples are designed to demonstrate
|
||||
the performance of different approaches to finding and downloading
|
||||
images from the Wikipedia. However, we don't want to hit the Wikipedia
|
||||
with multiple requests per second while testing, and we want to be
|
||||
able to simulate high latency and random network errors.
|
||||
|
||||
For this setup I chose **nginx** as the HTTP server because it is very
|
||||
fast and easy to configure, and the **vaurien** proxy because it was
|
||||
designed by Mozilla to introduce delays and network errors for testing.
|
||||
|
||||
The initial fixture data, ``docroot.zip``, contains a directory
|
||||
``docroot/Template-POTD/`` with 1096 small text files, each consisting
|
||||
of an HTML fragment (just a ``src="..."`` attribute) or an error message
|
||||
(for days when no picture was published, like 2013-09-12). These files
|
||||
correspond to every day of the years 2012, 2013 and 2014. The year 2012
|
||||
was a leap year, that's why there are 1096 files and not 1095.
|
||||
|
||||
Once these files are unpacked to the ``docroot/Template-POTD`` directory
|
||||
and **nginx** is configured, the ``build_fixture.py`` script can fetch the
|
||||
actual images from the Wikipedia for local storage in the directory
|
||||
``docroot/wikimedia/``.
|
||||
|
||||
When that is done you can configure **nginx** and **vaurien** to experiment
|
||||
with the ``daypicts*.py``examples without hitting the network.
|
||||
|
||||
|
||||
Instructions
|
||||
============
|
||||
|
||||
1. Unpack test data
|
||||
-------------------
|
||||
|
||||
Unpack the initial data in the ``fixture/`` directory and verify that 1096
|
||||
files were created in ``fixture/docroot/Template-POTD/``::
|
||||
|
||||
$ ls # inside the fixture/ directory
|
||||
README.rst docroot.zip
|
||||
$ unzip docroot.zip
|
||||
... many lines omitted...
|
||||
inflating: docroot/Template-POTD/2014-12-29
|
||||
inflating: docroot/Template-POTD/2014-12-30
|
||||
inflating: docroot/Template-POTD/2014-12-31
|
||||
$ ls docroot/Template-POTD/ | wc -w
|
||||
1096
|
||||
|
||||
|
||||
2. Install **nginx**
|
||||
--------------------
|
||||
|
||||
Download and install **nginx**. I used version 1.6.2 -- the latest
|
||||
stable version as I write this.
|
||||
|
||||
- Download page: http://nginx.org/en/download.html
|
||||
|
||||
- Beginner's guide: http://nginx.org/en/docs/beginners_guide.html
|
||||
|
||||
|
||||
3. Configure **nginx**
|
||||
----------------------
|
||||
|
||||
Edit the the ``nginx.conf`` file to set the port and document root.
|
||||
The file is usually found in ``/usr/local/nginx/conf``, ``/etc/nginx``,
|
||||
or ``/usr/local/etc/nginx``.
|
||||
|
||||
Most of the content in ``nginx.conf`` is within a block labeled ``http``
|
||||
and enclosed in curly braces. Within that block there can be multiple
|
||||
blocks labeled ``server``. Add another ``server`` block like this one::
|
||||
|
||||
server {
|
||||
listen 8001;
|
||||
|
||||
location / {
|
||||
root /full-path-to.../fixture/docroot;
|
||||
}
|
||||
}
|
||||
|
||||
After editing ``nginx.conf`` the server must be started (if it's not
|
||||
running) or told to reload the configuration file::
|
||||
|
||||
$ nginx # to start, if necessary
|
||||
$ nginx -s reload # to reload the configuration
|
||||
|
||||
To test the configuration, open the URL below in a browser. Doing so
|
||||
will download a small file named ``2014-01-01`` with an HTML fragment::
|
||||
|
||||
http://localhost:8001/Template-POTD/2014-01-01
|
||||
|
||||
If the test fails, please double check the procedure just described and
|
||||
refer to the **nginx** documentation.
|
||||
|
||||
|
||||
Platform-specific instructions
|
||||
==============================
|
||||
|
||||
Nginx setup on Mac OS X
|
||||
-----------------------
|
||||
|
||||
Homebrew (copy & paste code at the bottom of http://brew.sh/)::
|
||||
|
||||
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
|
||||
$ brew doctor
|
||||
$ brew install nginx
|
||||
|
||||
Download and unpack::
|
||||
|
||||
Docroot is: /usr/local/var/www
|
||||
/usr/local/etc/nginx/nginx.conf
|
||||
|
||||
To have launchd start nginx at login:
|
||||
ln -sfv /usr/local/opt/nginx/*.plist ~/Library/LaunchAgents
|
||||
Then to load nginx now:
|
||||
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.nginx.plist
|
||||
Or, if you don't want/need launchctl, you can just run:
|
||||
nginx
|
||||
|
||||
|
||||
|
||||
Nginx setup on Lubuntu 14.04.1 LTS
|
||||
----------------------------------
|
||||
|
||||
Docroot is: /usr/share/nginx/html
|
||||
|
@ -5,10 +5,10 @@ Configuring a local test environment
|
||||
tl;dr;
|
||||
======
|
||||
|
||||
This text explains how to configure *nginx* and *vaurien* to build
|
||||
a local mirror of the data to test the Wikipedia Picture of the Day
|
||||
examples while saving network bandwidth and introducing controlled
|
||||
delays and errors, thanks to the *vaurien* proxy.
|
||||
This text explains how to configure **nginx** and **vaurien** to build
|
||||
a local mirror of the data to run the Wikipedia Picture of the Day
|
||||
examples while avoiding network traffic and introducing controlled
|
||||
delays and errors for testing, thanks to the **vaurien** proxy.
|
||||
|
||||
|
||||
Rationale and overview
|
||||
@ -20,8 +20,8 @@ images from the Wikipedia. However, we don't want to hit the Wikipedia
|
||||
with multiple requests per second while testing, and we want to be
|
||||
able to simulate high latency and random network errors.
|
||||
|
||||
For this setup I chose *nginx* as the HTTP server because it is very
|
||||
fast and easy to configure, and the *vaurien* proxy because it was
|
||||
For this setup I chose **nginx** as the HTTP server because it is very
|
||||
fast and easy to configure, and the **vaurien** proxy because it was
|
||||
designed by Mozilla to introduce delays and network errors for testing.
|
||||
|
||||
The initial fixture data, ``docroot.zip``, contains a directory
|
||||
@ -32,22 +32,25 @@ correspond to every day of the years 2012, 2013 and 2014. The year 2012
|
||||
was a leap year, that's why there are 1096 files and not 1095.
|
||||
|
||||
Once these files are unpacked to the ``docroot/Template-POTD`` directory
|
||||
and *nginx* is configured, the ``build_fixture.py`` script can fetch the
|
||||
and **nginx** is configured, the ``build_fixture.py`` script can fetch the
|
||||
actual images from the Wikipedia for local storage in the directory
|
||||
``docroot/wikimedia/``.
|
||||
|
||||
When that is done you can configure *nginx* and *vaurien* to experiment
|
||||
When that is done you can configure **nginx** and **vaurien** to experiment
|
||||
with the ``daypicts*.py``examples without hitting the network.
|
||||
|
||||
|
||||
Instructions
|
||||
============
|
||||
|
||||
1. Unpack data
|
||||
--------------
|
||||
1. Unpack test data
|
||||
-------------------
|
||||
|
||||
Unpack the initial data and verify that 1096 files were created::
|
||||
Unpack the initial data in the ``fixture/`` directory and verify that 1096
|
||||
files were created in ``fixture/docroot/Template-POTD/``::
|
||||
|
||||
$ ls # inside the fixture/ directory
|
||||
README.rst docroot.zip
|
||||
$ unzip docroot.zip
|
||||
... many lines omitted...
|
||||
inflating: docroot/Template-POTD/2014-12-29
|
||||
@ -57,10 +60,10 @@ Unpack the initial data and verify that 1096 files were created::
|
||||
1096
|
||||
|
||||
|
||||
2. Install *nginx*
|
||||
------------------
|
||||
2. Install **nginx**
|
||||
--------------------
|
||||
|
||||
Download and install *nginx*. I used version 1.6.2 -- the latest
|
||||
Download and install **nginx**. I used version 1.6.2 -- the latest
|
||||
stable version as I write this.
|
||||
|
||||
- Download page: http://nginx.org/en/download.html
|
||||
@ -68,8 +71,8 @@ stable version as I write this.
|
||||
- Beginner's guide: http://nginx.org/en/docs/beginners_guide.html
|
||||
|
||||
|
||||
3. Configure *nginx*
|
||||
--------------------
|
||||
3. Configure **nginx**
|
||||
----------------------
|
||||
|
||||
Edit the the ``nginx.conf`` file to set the port and document root.
|
||||
The file is usually found in ``/usr/local/nginx/conf``, ``/etc/nginx``,
|
||||
@ -83,10 +86,20 @@ blocks labeled ``server``. Add another ``server`` block like this one::
|
||||
listen 8001;
|
||||
|
||||
location / {
|
||||
root /full-path-to-your-directory/fixture/docroot;
|
||||
root /full-path-to.../fixture/docroot;
|
||||
}
|
||||
}
|
||||
|
||||
After editing ``nginx.conf`` the server must be started (if it's not
|
||||
running) or told to reload the configuration file::
|
||||
|
||||
$ nginx # to start, if necessary
|
||||
$ nginx -s reload # to reload the configuration
|
||||
|
||||
To test the configuration, open the URL below in a browser. Doing so
|
||||
will download a small file named ``2014-01-01`` with an HTML fragment::
|
||||
|
||||
http://localhost:8001/Template-POTD/2014-01-01
|
||||
|
||||
If the test fails, please double check the procedure just described and
|
||||
refer to the **nginx** documentation.
|
||||
|
39
concurrency/wikipedia/orig/README.rst
Normal file
39
concurrency/wikipedia/orig/README.rst
Normal file
@ -0,0 +1,39 @@
|
||||
=====================================
|
||||
Wikipedia Picture of the Day examples
|
||||
=====================================
|
||||
|
||||
These examples use various asynchronous programming techniques to download
|
||||
images and metadata from the English Wikipedia `Picture of the Day`_ archive.
|
||||
|
||||
.. _Picture of the Day: http://en.wikipedia.org/wiki/Wikipedia:Picture_of_the_day/Archive
|
||||
|
||||
|
||||
--------
|
||||
Timings
|
||||
--------
|
||||
|
||||
``sync.py``
|
||||
===========
|
||||
|
||||
::
|
||||
|
||||
$ time python sync.py 2014-06 -q 5
|
||||
5 images downloaded (167.8 Kbytes total)
|
||||
|
||||
real 0m6.272s
|
||||
user 0m0.065s
|
||||
sys 0m0.039s
|
||||
|
||||
$ time python sync.py 2014-06 -q 5
|
||||
5 images downloaded (167.8 Kbytes total)
|
||||
|
||||
real 0m5.447s
|
||||
user 0m0.068s
|
||||
sys 0m0.040s
|
||||
|
||||
$ time python sync.py 2014-06 -q 5
|
||||
5 images downloaded (167.8 Kbytes total)
|
||||
|
||||
real 0m6.314s
|
||||
user 0m0.068s
|
||||
sys 0m0.040s
|
BIN
futures/charfinder/charfinder_index.pickle
Normal file
BIN
futures/charfinder/charfinder_index.pickle
Normal file
Binary file not shown.
194
futures/countries/README.rst
Normal file
194
futures/countries/README.rst
Normal file
@ -0,0 +1,194 @@
|
||||
====================================
|
||||
Configuring a local test environment
|
||||
====================================
|
||||
|
||||
tl;dr;
|
||||
======
|
||||
|
||||
This text explains how to configure **nginx** and **vaurien** to build a local
|
||||
mirror of the data to run the flag download examples while avoiding network
|
||||
traffic and introducing controlled delays and errors for testing, thanks to
|
||||
the **vaurien** proxy.
|
||||
|
||||
|
||||
Rationale and overview
|
||||
======================
|
||||
|
||||
The flag download examples are designed to compare the performance of
|
||||
different approaches to finding and downloading files from the Web. However,
|
||||
we don't want to hit a public server with multiple requests per second while
|
||||
testing, and we want to be able to simulate high latency and random network
|
||||
errors.
|
||||
|
||||
For this setup I chose **nginx** as the HTTP server because it is very fast
|
||||
and easy to configure, and the **vaurien** proxy because it was designed by
|
||||
Mozilla to introduce delays and network errors for testing.
|
||||
|
||||
The archive ``flags.zip``, contains a directory ``flags/`` with 194
|
||||
subdirectories, each containing a ``.gif` image and a ``metadata.json`` file.
|
||||
These images are public-domain flags copied from the CIA World Fact Book [1].
|
||||
|
||||
[1] https://www.cia.gov/library/publications/the-world-factbook/
|
||||
|
||||
Once these files are unpacked to the ``flags/`` directory and **nginx** is
|
||||
configured, you can experiment with the ``flags*.py``examples without hitting
|
||||
the network.
|
||||
|
||||
|
||||
Instructions
|
||||
============
|
||||
|
||||
1. Unpack test data
|
||||
-------------------
|
||||
|
||||
Unpack the initial data in the ``countries/`` directory and verify that 194
|
||||
directories are created in ``countries/flags/``, each with a ``.gif`` and
|
||||
a ``metadata.json`` file::
|
||||
|
||||
$ unzip flags.zip
|
||||
... many lines omitted...
|
||||
creating: flags/zw/
|
||||
inflating: flags/zw/metadata.json
|
||||
inflating: flags/zw/zw.gif
|
||||
$ ls flags | wc -w
|
||||
194
|
||||
$ find flags | grep .gif | wc -l
|
||||
194
|
||||
$ find flags | grep .json | wc -l
|
||||
194
|
||||
$ ls flags/ad
|
||||
ad.gif metadata.json
|
||||
|
||||
|
||||
2. Install **nginx**
|
||||
--------------------
|
||||
|
||||
Download and install **nginx**. I used version 1.6.2 -- the latest
|
||||
stable version as I write this.
|
||||
|
||||
- Download page: http://nginx.org/en/download.html
|
||||
|
||||
- Beginner's guide: http://nginx.org/en/docs/beginners_guide.html
|
||||
|
||||
|
||||
3. Configure **nginx**
|
||||
----------------------
|
||||
|
||||
Edit the the ``nginx.conf`` file to set the port and document root.
|
||||
You can determine which ``nginx.conf`` is in use by running::
|
||||
|
||||
$ nginx -V
|
||||
|
||||
The output starts with::
|
||||
|
||||
nginx version: nginx/1.6.2
|
||||
built by clang 6.0 (clang-600.0.51) (based on LLVM 3.5svn)
|
||||
TLS SNI support enabled
|
||||
configure arguments:...
|
||||
|
||||
Among the configure arguments you'll see ``--conf-path=``. That's the
|
||||
file you will edit.
|
||||
|
||||
Most of the content in ``nginx.conf`` is within a block labeled ``http``
|
||||
and enclosed in curly braces. Within that block there can be multiple
|
||||
blocks labeled ``server``. Add another ``server`` block like this one::
|
||||
|
||||
server {
|
||||
listen 8001;
|
||||
|
||||
location /flags/ {
|
||||
root /full-path-to.../countries/;
|
||||
}
|
||||
}
|
||||
|
||||
After editing ``nginx.conf`` the server must be started (if it's not
|
||||
running) or told to reload the configuration file::
|
||||
|
||||
$ nginx # to start, if necessary
|
||||
$ nginx -s reload # to reload the configuration
|
||||
|
||||
To test the configuration, open the URL below in a browser. You should
|
||||
see the blue, yellow and red flag of Andorra::
|
||||
|
||||
http://localhost:8001/flags/ad/ad.gif
|
||||
|
||||
If the test fails, please double check the procedure just described and
|
||||
refer to the **nginx** documentation.
|
||||
|
||||
At this point you may run the ``flags_*2.py`` examples against the **nginx**
|
||||
install by changing the ``BASE_URL`` constant in ``flags_sequential2.py``.
|
||||
However, **nginx** is so fast that you will not see much difference in run
|
||||
time between the sequential and the threaded versions, for example. For more
|
||||
realistic testing with simulated network lag, we need **vaurien**.
|
||||
|
||||
|
||||
4. Install and run **vaurien**
|
||||
------------------------------
|
||||
|
||||
**vaurien depends on gevent which is only available for Python 2.5-2.7. To
|
||||
install vaurien I opened another shell, created another ``virtualenv`` for
|
||||
Python 2.7, and used that environment to install and run vaurien::
|
||||
|
||||
$ virtualenv-2.7 .env27 --no-site-packages --distribute
|
||||
New python executable in .env27/bin/python
|
||||
Installing setuptools, pip...done.
|
||||
$ . .env27/bin/activate
|
||||
(.env27)$ pip install vaurien
|
||||
Downloading/unpacking vaurien
|
||||
Downloading vaurien-1.9.tar.gz (50kB): 50kB downloaded
|
||||
...many lines and a few minutes later...
|
||||
|
||||
Successfully installed vaurien cornice gevent statsd-client vaurienclient
|
||||
greenlet http-parser pyramid simplejson requests zope.interface
|
||||
translationstring PasteDeploy WebOb repoze.lru zope.deprecation venusian
|
||||
Cleaning up...
|
||||
(.env27)$
|
||||
|
||||
Using that same shell with the ``.env27`` activated, run the ``vaurien_delay.sh`` script in the ``countries/`` directory::
|
||||
|
||||
(.env27)$ $ ./vaurien_delay.sh
|
||||
2015-02-25 20:20:17 [69124] [INFO] Starting the Chaos TCP Server
|
||||
2015-02-25 20:20:17 [69124] [INFO] Options:
|
||||
2015-02-25 20:20:17 [69124] [INFO] * proxies from localhost:8002 to localhost:8001
|
||||
2015-02-25 20:20:17 [69124] [INFO] * timeout: 30
|
||||
2015-02-25 20:20:17 [69124] [INFO] * stay_connected: 0
|
||||
2015-02-25 20:20:17 [69124] [INFO] * pool_max_size: 100
|
||||
2015-02-25 20:20:17 [69124] [INFO] * pool_timeout: 30
|
||||
2015-02-25 20:20:17 [69124] [INFO] * async_mode: 1
|
||||
|
||||
The ``vaurien_delay.sh`` adds a 1s delay to every response.
|
||||
|
||||
There is also the ``vaurien_error_delay.sh`` script which produces errors in 25% of the responses and a .5 se delay to 50% of the responses.
|
||||
|
||||
|
||||
Platform-specific instructions
|
||||
==============================
|
||||
|
||||
Nginx setup on Mac OS X
|
||||
-----------------------
|
||||
|
||||
Homebrew (copy & paste code at the bottom of http://brew.sh/)::
|
||||
|
||||
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
|
||||
$ brew doctor
|
||||
$ brew install nginx
|
||||
|
||||
Download and unpack::
|
||||
|
||||
Docroot is: /usr/local/var/www
|
||||
/usr/local/etc/nginx/nginx.conf
|
||||
|
||||
To have launchd start nginx at login:
|
||||
ln -sfv /usr/local/opt/nginx/*.plist ~/Library/LaunchAgents
|
||||
Then to load nginx now:
|
||||
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.nginx.plist
|
||||
Or, if you don't want/need launchctl, you can just run:
|
||||
nginx
|
||||
|
||||
|
||||
|
||||
Nginx setup on Lubuntu 14.04.1 LTS
|
||||
----------------------------------
|
||||
|
||||
Docroot is: /usr/share/nginx/html
|
||||
|
@ -6,5 +6,5 @@ __pycache__
|
||||
.gitignore
|
||||
.DS_Store
|
||||
*.arr
|
||||
README.rst
|
||||
/README.rst
|
||||
LICENSE
|
||||
|
9
support/isis2json/README.rst
Normal file
9
support/isis2json/README.rst
Normal file
@ -0,0 +1,9 @@
|
||||
isis2json.py
|
||||
============
|
||||
|
||||
This directory contains the ``isis2json.py`` script discussed in
|
||||
Chapter 14 of Fluent Python, with minimal dependencies, just to
|
||||
support the generation of the book listing.
|
||||
|
||||
The main repository for this script is:
|
||||
https://github.com/fluentpython/isis2json
|
Loading…
Reference in New Issue
Block a user