195 lines
6.3 KiB
ReStructuredText
195 lines
6.3 KiB
ReStructuredText
====================================
|
|
Configuring a local test environment
|
|
====================================
|
|
|
|
tl;dr;
|
|
======
|
|
|
|
This text explains how to configure **nginx** and **vaurien** to build a local
|
|
mirror of the data to run the flag download examples while avoiding network
|
|
traffic and introducing controlled delays and errors for testing, thanks to
|
|
the **vaurien** proxy.
|
|
|
|
|
|
Rationale and overview
|
|
======================
|
|
|
|
The flag download examples are designed to compare the performance of
|
|
different approaches to finding and downloading files from the Web. However,
|
|
we don't want to hit a public server with multiple requests per second while
|
|
testing, and we want to be able to simulate high latency and random network
|
|
errors.
|
|
|
|
For this setup I chose **nginx** as the HTTP server because it is very fast
|
|
and easy to configure, and the **vaurien** proxy because it was designed by
|
|
Mozilla to introduce delays and network errors for testing.
|
|
|
|
The archive ``flags.zip``, contains a directory ``flags/`` with 194
|
|
subdirectories, each containing a ``.gif` image and a ``metadata.json`` file.
|
|
These images are public-domain flags copied from the CIA World Fact Book [1].
|
|
|
|
[1] https://www.cia.gov/library/publications/the-world-factbook/
|
|
|
|
Once these files are unpacked to the ``flags/`` directory and **nginx** is
|
|
configured, you can experiment with the ``flags*.py``examples without hitting
|
|
the network.
|
|
|
|
|
|
Instructions
|
|
============
|
|
|
|
1. Unpack test data
|
|
-------------------
|
|
|
|
Unpack the initial data in the ``countries/`` directory and verify that 194
|
|
directories are created in ``countries/flags/``, each with a ``.gif`` and
|
|
a ``metadata.json`` file::
|
|
|
|
$ unzip flags.zip
|
|
... many lines omitted...
|
|
creating: flags/zw/
|
|
inflating: flags/zw/metadata.json
|
|
inflating: flags/zw/zw.gif
|
|
$ ls flags | wc -w
|
|
194
|
|
$ find flags | grep .gif | wc -l
|
|
194
|
|
$ find flags | grep .json | wc -l
|
|
194
|
|
$ ls flags/ad
|
|
ad.gif metadata.json
|
|
|
|
|
|
2. Install **nginx**
|
|
--------------------
|
|
|
|
Download and install **nginx**. I used version 1.6.2 -- the latest
|
|
stable version as I write this.
|
|
|
|
- Download page: http://nginx.org/en/download.html
|
|
|
|
- Beginner's guide: http://nginx.org/en/docs/beginners_guide.html
|
|
|
|
|
|
3. Configure **nginx**
|
|
----------------------
|
|
|
|
Edit the the ``nginx.conf`` file to set the port and document root.
|
|
You can determine which ``nginx.conf`` is in use by running::
|
|
|
|
$ nginx -V
|
|
|
|
The output starts with::
|
|
|
|
nginx version: nginx/1.6.2
|
|
built by clang 6.0 (clang-600.0.51) (based on LLVM 3.5svn)
|
|
TLS SNI support enabled
|
|
configure arguments:...
|
|
|
|
Among the configure arguments you'll see ``--conf-path=``. That's the
|
|
file you will edit.
|
|
|
|
Most of the content in ``nginx.conf`` is within a block labeled ``http``
|
|
and enclosed in curly braces. Within that block there can be multiple
|
|
blocks labeled ``server``. Add another ``server`` block like this one::
|
|
|
|
server {
|
|
listen 8001;
|
|
|
|
location /flags/ {
|
|
root /full-path-to.../countries/;
|
|
}
|
|
}
|
|
|
|
After editing ``nginx.conf`` the server must be started (if it's not
|
|
running) or told to reload the configuration file::
|
|
|
|
$ nginx # to start, if necessary
|
|
$ nginx -s reload # to reload the configuration
|
|
|
|
To test the configuration, open the URL below in a browser. You should
|
|
see the blue, yellow and red flag of Andorra::
|
|
|
|
http://localhost:8001/flags/ad/ad.gif
|
|
|
|
If the test fails, please double check the procedure just described and
|
|
refer to the **nginx** documentation.
|
|
|
|
At this point you may run the ``flags_*2.py`` examples against the **nginx**
|
|
install by changing the ``BASE_URL`` constant in ``flags_sequential2.py``.
|
|
However, **nginx** is so fast that you will not see much difference in run
|
|
time between the sequential and the threaded versions, for example. For more
|
|
realistic testing with simulated network lag, we need **vaurien**.
|
|
|
|
|
|
4. Install and run **vaurien**
|
|
------------------------------
|
|
|
|
**vaurien depends on gevent which is only available for Python 2.5-2.7. To
|
|
install vaurien I opened another shell, created another ``virtualenv`` for
|
|
Python 2.7, and used that environment to install and run vaurien::
|
|
|
|
$ virtualenv-2.7 .env27 --no-site-packages --distribute
|
|
New python executable in .env27/bin/python
|
|
Installing setuptools, pip...done.
|
|
$ . .env27/bin/activate
|
|
(.env27)$ pip install vaurien
|
|
Downloading/unpacking vaurien
|
|
Downloading vaurien-1.9.tar.gz (50kB): 50kB downloaded
|
|
...many lines and a few minutes later...
|
|
|
|
Successfully installed vaurien cornice gevent statsd-client vaurienclient
|
|
greenlet http-parser pyramid simplejson requests zope.interface
|
|
translationstring PasteDeploy WebOb repoze.lru zope.deprecation venusian
|
|
Cleaning up...
|
|
(.env27)$
|
|
|
|
Using that same shell with the ``.env27`` activated, run the ``vaurien_delay.sh`` script in the ``countries/`` directory::
|
|
|
|
(.env27)$ $ ./vaurien_delay.sh
|
|
2015-02-25 20:20:17 [69124] [INFO] Starting the Chaos TCP Server
|
|
2015-02-25 20:20:17 [69124] [INFO] Options:
|
|
2015-02-25 20:20:17 [69124] [INFO] * proxies from localhost:8002 to localhost:8001
|
|
2015-02-25 20:20:17 [69124] [INFO] * timeout: 30
|
|
2015-02-25 20:20:17 [69124] [INFO] * stay_connected: 0
|
|
2015-02-25 20:20:17 [69124] [INFO] * pool_max_size: 100
|
|
2015-02-25 20:20:17 [69124] [INFO] * pool_timeout: 30
|
|
2015-02-25 20:20:17 [69124] [INFO] * async_mode: 1
|
|
|
|
The ``vaurien_delay.sh`` adds a 1s delay to every response.
|
|
|
|
There is also the ``vaurien_error_delay.sh`` script which produces errors in 25% of the responses and a .5 se delay to 50% of the responses.
|
|
|
|
|
|
Platform-specific instructions
|
|
==============================
|
|
|
|
Nginx setup on Mac OS X
|
|
-----------------------
|
|
|
|
Homebrew (copy & paste code at the bottom of http://brew.sh/)::
|
|
|
|
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
|
|
$ brew doctor
|
|
$ brew install nginx
|
|
|
|
Download and unpack::
|
|
|
|
Docroot is: /usr/local/var/www
|
|
/usr/local/etc/nginx/nginx.conf
|
|
|
|
To have launchd start nginx at login:
|
|
ln -sfv /usr/local/opt/nginx/*.plist ~/Library/LaunchAgents
|
|
Then to load nginx now:
|
|
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.nginx.plist
|
|
Or, if you don't want/need launchctl, you can just run:
|
|
nginx
|
|
|
|
|
|
|
|
Nginx setup on Lubuntu 14.04.1 LTS
|
|
----------------------------------
|
|
|
|
Docroot is: /usr/share/nginx/html
|
|
|