daviddoji/example-code-2e
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

removed 1st edition code

Browse Source
This commit is contained in:
Luciano Ramalho
2020-03-11 18:03:55 -03:00
parent 1be97eea3e
commit 506d343843
460 changed files with 0 additions and 108428 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
20-futures/README.rst
View File

@@ -1,4 +0,0 @@
Sample code for Chapter 17 - "Concurrency with futures"
From the book "Fluent Python" by Luciano Ramalho (O'Reilly, 2015)
http://shop.oreilly.com/product/0636920032519.do

194
20-futures/countries/README.rst
View File

@@ -1,194 +0,0 @@
============================
Setting up Nginx and Vaurien
============================
This text explains how to configure Nginx and Vaurien to test HTTP client code while avoiding network traffic and introducing simulated delays and errors. This setup is necessary if you want to experiment with the ``flags2*.py`` image download examples in this directory -- covered in chapters 17 and 18 of Fluent Python.
Overview
========
The flag download examples are designed to compare the performance of different approaches to finding and downloading files from the Web. However, we don't want to hit a public server with multiple requests per second while testing, and we want to be able to simulate high latency and random network errors.
For this setup I chose Nginx as the HTTP server because it is very fast and easy to configure, and the Vaurien proxy because it was designed by Mozilla to introduce delays and network errors for testing Web services.
The archive ``flags.zip``, contains a directory ``flags/`` with 194 subdirectories, each containing a ``.gif`` image and a ``metadata.json`` file. These are public-domain images copied from the `CIA World Fact Book <https://www.cia.gov/library/publications/the-world-factbook/>`_.
Once these files are unpacked to the ``flags/`` directory and Nginx is configured, you can experiment with the ``flags2*.py`` examples without hitting the network.
Procedure
=========
1. Unpack test data
-------------------
The instructions in this section are for GNU/Linux or OSX using the command line. Windows users should have no difficulty doing the same operations with the Windows Exporer GUI.
Unpack the initial data in the ``countries/`` directory::
$ unzip flags.zip
... many lines omitted ...
creating: flags/zw/
inflating: flags/zw/metadata.json
inflating: flags/zw/zw.gif
Verify that 194 directories are created in ``countries/flags/``, each with a ``.gif`` and a ``metadata.json`` file::
$ ls flags | wc -w
194
$ find flags | grep .gif | wc -l
194
$ find flags | grep .json | wc -l
194
$ ls flags/ad
ad.gif metadata.json
2. Install Nginx
----------------
Download and install Nginx. I used version 1.6.2 -- the latest stable version as I write this.
* Download page: http://nginx.org/en/download.html
* Beginner's guide: http://nginx.org/en/docs/beginners_guide.html
3. Configure Nginx
------------------
Edit the the ``nginx.conf`` file to set the port and document root. You can determine which ``nginx.conf`` is in use by running::
$ nginx -V
The output starts with::
nginx version: nginx/1.6.2
built by clang 6.0 (clang-600.0.51) (based on LLVM 3.5svn)
TLS SNI support enabled
configure arguments:...
Among the configure arguments you'll see ``--conf-path=``. That's the file you will edit.
Most of the content in ``nginx.conf`` is within a block labeled ``http`` and enclosed in curly braces. Within that block there can be multiple blocks labeled ``server``. Add another ``server`` block like this one::
server {
listen 8001;
location /flags/ {
root /full-path-to.../countries/;
}
}
After editing ``nginx.conf`` the server must be started (if it's not running) or told to reload the configuration file::
$ nginx # to start, if necessary
$ nginx -s reload # to reload the configuration
To test the configuration, open the URL http://localhost:8001/flags/ad/ad.gif in a browser. You should see the blue, yellow and red flag of Andorra.
If the test fails, please double check the procedure just described and refer to the Nginx documentation.
At this point you may run the ``flags_*2.py`` examples against the Nginx install by providing the ``--server LOCAL`` command line option. For example::
$ python3 flags2_threadpool.py -s LOCAL
LOCAL site: http://localhost:8001/flags
Searching for 20 flags: from BD to VN
20 concurrent connections will be used.
--------------------
20 flags downloaded.
Elapsed time: 0.09s
Note that Nginx is so fast that you will not see much difference in run time between the sequential and the concurrent versions. For more realistic testing with simulated network lag, we need to set up the Vaurien proxy.
4. Install and run Vaurien
--------------------------
Vaurien depends on gevent which is only available for Python 2.5-2.7. To install Vaurien I opened another shell, created a virtualenv for Python 2.7, and used that environment to install and run the ``vaurien`` package::
$ virtualenv-2.7 .env27 --no-site-packages --distribute
New python executable in .env27/bin/python
Installing setuptools, pip...done.
$ . .env27/bin/activate
(.env27)$ pip install vaurien
Downloading/unpacking vaurien
Downloading vaurien-1.9.tar.gz (50kB): 50kB downloaded
...many lines and a few minutes later...
Successfully installed vaurien cornice gevent statsd-client vaurienclient
greenlet http-parser pyramid simplejson requests zope.interface
translationstring PasteDeploy WebOb repoze.lru zope.deprecation venusian
Cleaning up...
Using that same shell with the ``.env27`` environment active, run the ``vaurien_delay.sh`` script in the ``countries/`` directory::
(.env27)$ $ ./vaurien_delay.sh
2015-02-25 20:20:17 [69124] [INFO] Starting the Chaos TCP Server
2015-02-25 20:20:17 [69124] [INFO] Options:
2015-02-25 20:20:17 [69124] [INFO] * proxies from localhost:8002 to localhost:8001
2015-02-25 20:20:17 [69124] [INFO] * timeout: 30
2015-02-25 20:20:17 [69124] [INFO] * stay_connected: 0
2015-02-25 20:20:17 [69124] [INFO] * pool_max_size: 100
2015-02-25 20:20:17 [69124] [INFO] * pool_timeout: 30
2015-02-25 20:20:17 [69124] [INFO] * async_mode: 1
The ``vaurien_delay.sh`` creates an HTTP proxy on port 8002 which adds a 1s delay to every response. You can test it with a browser on port 8002: http://localhost:8002/flags/ad/ad.gif -- the flag of Andorra should appear after 1 second.
There is also the ``vaurien_error_delay.sh`` script which runs a proxy on port 8003 producing errors in 25% of the responses and a .5 s delay to 50% of the responses. You can also test it with the browser on port 8003, but rememeber that errors are expected.
Platform-specific instructions
==============================
Nginx setup on Mac OS X
------------------------
Homebrew (copy & paste code at the bottom of http://brew.sh/)::
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew doctor
$ brew install nginx
Download and unpack::
Docroot is: /usr/local/var/www
/usr/local/etc/nginx/nginx.conf
::
To have launchd start nginx at login:
ln -sfv /usr/local/opt/nginx/*.plist ~/Library/LaunchAgents
Then to load nginx now:
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.nginx.plist
Or, if you don't want/need launchctl, you can just run:
nginx
Nginx setup on Lubuntu 14.04.1 LTS
----------------------------------
Docroot is: /usr/share/nginx/html

8
20-futures/countries/country_codes.txt
View File

@@ -1,8 +0,0 @@
AD AE AF AG AL AM AO AR AT AU AZ BA BB BD BE BF BG BH BI BJ BN BO BR BS BT
BW BY BZ CA CD CF CG CH CI CL CM CN CO CR CU CV CY CZ DE DJ DK DM DZ EC EE
EG ER ES ET FI FJ FM FR GA GB GD GE GH GM GN GQ GR GT GW GY HN HR HT HU ID
IE IL IN IQ IR IS IT JM JO JP KE KG KH KI KM KN KP KR KW KZ LA LB LC LI LK
LR LS LT LU LV LY MA MC MD ME MG MH MK ML MM MN MR MT MU MV MW MX MY MZ NA
NE NG NI NL NO NP NR NZ OM PA PE PG PH PK PL PT PW PY QA RO RS RU RW SA SB
SC SD SE SG SI SK SL SM SN SO SR SS ST SV SY SZ TD TG TH TJ TL TM TN TO TR
TT TV TW TZ UA UG US UY UZ VA VC VE VN VU WS YE ZA ZM ZW

63
20-futures/countries/flags.py
View File

@@ -1,63 +0,0 @@
"""Download flags of top 20 countries by population
Sequential version
Sample run::
$ python3 flags.py
BD BR CD CN DE EG ET FR ID IN IR JP MX NG PH PK RU TR US VN
20 flags downloaded in 10.16s
"""
# BEGIN FLAGS_PY
import os
import time
import sys
import requests # <1>
POP20_CC = ('CN IN US ID BR PK NG BD RU JP '
'MX PH VN ET EG DE IR TR CD FR').split() # <2>
BASE_URL = 'http://flupy.org/data/flags' # <3>
DEST_DIR = 'downloads/' # <4>
def save_flag(img, filename): # <5>
path = os.path.join(DEST_DIR, filename)
with open(path, 'wb') as fp:
fp.write(img)
def get_flag(cc): # <6>
url = '{}/{cc}/{cc}.gif'.format(BASE_URL, cc=cc.lower())
resp = requests.get(url)
return resp.content
def show(text): # <7>
print(text, end=' ')
sys.stdout.flush()
def download_many(cc_list): # <8>
for cc in sorted(cc_list): # <9>
image = get_flag(cc)
show(cc)
save_flag(image, cc.lower() + '.gif')
return len(cc_list)
def main(download_many): # <10>
t0 = time.time()
count = download_many(POP20_CC)
elapsed = time.time() - t0
msg = '\n{} flags downloaded in {:.2f}s'
print(msg.format(count, elapsed))
if __name__ == '__main__':
main(download_many) # <11>
# END FLAGS_PY

BIN
20-futures/countries/flags.zip
View File

Binary file not shown.

120
20-futures/countries/flags2_asyncio.py
View File

@@ -1,120 +0,0 @@
"""Download flags of countries (with error handling).
asyncio yield-from version
Sample run::
$ python3 flags2_asyncio.py -s ERROR -e -m 200
ERROR site: http://localhost:8003/flags
Searching for 676 flags: from AA to ZZ
200 concurrent connections will be used.
--------------------
146 flags downloaded.
363 not found.
167 errors.
Elapsed time: 2.59s
"""
# BEGIN FLAGS2_ASYNCIO_TOP
import asyncio
import collections
import contextlib
import aiohttp
from aiohttp import web
import tqdm
from flags2_common import main, HTTPStatus, Result, save_flag
# default set low to avoid errors from remote site, such as
# 503 - Service Temporarily Unavailable
DEFAULT_CONCUR_REQ = 5
MAX_CONCUR_REQ = 1000
class FetchError(Exception): # <1>
def __init__(self, country_code):
self.country_code = country_code
@asyncio.coroutine
def get_flag(base_url, cc): # <2>
url = '{}/{cc}/{cc}.gif'.format(base_url, cc=cc.lower())
resp = yield from aiohttp.request('GET', url)
with contextlib.closing(resp):
if resp.status == 200:
image = yield from resp.read()
return image
elif resp.status == 404:
raise web.HTTPNotFound()
else:
raise aiohttp.HttpProcessingError(
code=resp.status, message=resp.reason,
headers=resp.headers)
@asyncio.coroutine
def download_one(cc, base_url, semaphore, verbose): # <3>
try:
with (yield from semaphore): # <4>
image = yield from get_flag(base_url, cc) # <5>
except web.HTTPNotFound: # <6>
status = HTTPStatus.not_found
msg = 'not found'
except Exception as exc:
raise FetchError(cc) from exc # <7>
else:
save_flag(image, cc.lower() + '.gif') # <8>
status = HTTPStatus.ok
msg = 'OK'
if verbose and msg:
print(cc, msg)
return Result(status, cc)
# END FLAGS2_ASYNCIO_TOP
# BEGIN FLAGS2_ASYNCIO_DOWNLOAD_MANY
@asyncio.coroutine
def downloader_coro(cc_list, base_url, verbose, concur_req): # <1>
counter = collections.Counter()
semaphore = asyncio.Semaphore(concur_req) # <2>
to_do = [download_one(cc, base_url, semaphore, verbose)
for cc in sorted(cc_list)] # <3>
to_do_iter = asyncio.as_completed(to_do) # <4>
if not verbose:
to_do_iter = tqdm.tqdm(to_do_iter, total=len(cc_list)) # <5>
for future in to_do_iter: # <6>
try:
res = yield from future # <7>
except FetchError as exc: # <8>
country_code = exc.country_code # <9>
try:
error_msg = exc.__cause__.args[0] # <10>
except IndexError:
error_msg = exc.__cause__.__class__.__name__ # <11>
if verbose and error_msg:
msg = '*** Error for {}: {}'
print(msg.format(country_code, error_msg))
status = HTTPStatus.error
else:
status = res.status
counter[status] += 1 # <12>
return counter # <13>
def download_many(cc_list, base_url, verbose, concur_req):
loop = asyncio.get_event_loop()
coro = downloader_coro(cc_list, base_url, verbose, concur_req)
counts = loop.run_until_complete(coro) # <14>
loop.close() # <15>
return counts
if __name__ == '__main__':
main(download_many, DEFAULT_CONCUR_REQ, MAX_CONCUR_REQ)
# END FLAGS2_ASYNCIO_DOWNLOAD_MANY

109
20-futures/countries/flags2_asyncio_executor.py
View File

@@ -1,109 +0,0 @@
"""Download flags of countries (with error handling).
asyncio version using thread pool to save files
"""
import asyncio
import collections
import contextlib
import aiohttp
from aiohttp import web
import tqdm
from flags2_common import main, HTTPStatus, Result, save_flag
# default set low to avoid errors from remote site, such as
# 503 - Service Temporarily Unavailable
DEFAULT_CONCUR_REQ = 5
MAX_CONCUR_REQ = 1000
class FetchError(Exception):
def __init__(self, country_code):
self.country_code = country_code
@asyncio.coroutine
def get_flag(base_url, cc):
url = '{}/{cc}/{cc}.gif'.format(base_url, cc=cc.lower())
resp = yield from aiohttp.request('GET', url)
with contextlib.closing(resp):
if resp.status == 200:
image = yield from resp.read()
return image
elif resp.status == 404:
raise web.HTTPNotFound()
else:
raise aiohttp.HttpProcessingError(
code=resp.status, message=resp.reason,
headers=resp.headers)
# BEGIN FLAGS2_ASYNCIO_EXECUTOR
@asyncio.coroutine
def download_one(cc, base_url, semaphore, verbose):
try:
with (yield from semaphore):
image = yield from get_flag(base_url, cc)
except web.HTTPNotFound:
status = HTTPStatus.not_found
msg = 'not found'
except Exception as exc:
raise FetchError(cc) from exc
else:
loop = asyncio.get_event_loop() # <1>
loop.run_in_executor(None, # <2>
save_flag, image, cc.lower() + '.gif') # <3>
status = HTTPStatus.ok
msg = 'OK'
if verbose and msg:
print(cc, msg)
return Result(status, cc)
# END FLAGS2_ASYNCIO_EXECUTOR
@asyncio.coroutine
def downloader_coro(cc_list, base_url, verbose, concur_req):
counter = collections.Counter()
semaphore = asyncio.Semaphore(concur_req)
to_do = [download_one(cc, base_url, semaphore, verbose)
for cc in sorted(cc_list)]
to_do_iter = asyncio.as_completed(to_do)
if not verbose:
to_do_iter = tqdm.tqdm(to_do_iter, total=len(cc_list))
for future in to_do_iter:
try:
res = yield from future
except FetchError as exc:
country_code = exc.country_code
try:
error_msg = exc.__cause__.args[0]
except IndexError:
error_msg = exc.__cause__.__class__.__name__
if verbose and error_msg:
msg = '*** Error for {}: {}'
print(msg.format(country_code, error_msg))
status = HTTPStatus.error
else:
status = res.status
counter[status] += 1
return counter
def download_many(cc_list, base_url, verbose, concur_req):
loop = asyncio.get_event_loop()
coro = downloader_coro(cc_list, base_url, verbose, concur_req)
counts = loop.run_until_complete(coro)
loop.close()
return counts
if __name__ == '__main__':
main(download_many, DEFAULT_CONCUR_REQ, MAX_CONCUR_REQ)

104
20-futures/countries/flags2_await.py
View File

@@ -1,104 +0,0 @@
"""Download flags of countries (with error handling).
asyncio async/await version
"""
# BEGIN FLAGS2_ASYNCIO_TOP
import asyncio
import collections
from contextlib import closing
import aiohttp
from aiohttp import web
import tqdm
from flags2_common import main, HTTPStatus, Result, save_flag
# default set low to avoid errors from remote site, such as
# 503 - Service Temporarily Unavailable
DEFAULT_CONCUR_REQ = 5
MAX_CONCUR_REQ = 1000
class FetchError(Exception): # <1>
def __init__(self, country_code):
self.country_code = country_code
async def get_flag(base_url, cc): # <2>
url = '{}/{cc}/{cc}.gif'.format(base_url, cc=cc.lower())
with closing(await aiohttp.request('GET', url)) as resp:
if resp.status == 200:
image = await resp.read()
return image
elif resp.status == 404:
raise web.HTTPNotFound()
else:
raise aiohttp.HttpProcessingError(
code=resp.status, message=resp.reason,
headers=resp.headers)
async def download_one(cc, base_url, semaphore, verbose): # <3>
try:
with (await semaphore): # <4>
image = await get_flag(base_url, cc) # <5>
except web.HTTPNotFound: # <6>
status = HTTPStatus.not_found
msg = 'not found'
except Exception as exc:
raise FetchError(cc) from exc # <7>
else:
save_flag(image, cc.lower() + '.gif') # <8>
status = HTTPStatus.ok
msg = 'OK'
if verbose and msg:
print(cc, msg)
return Result(status, cc)
# END FLAGS2_ASYNCIO_TOP
# BEGIN FLAGS2_ASYNCIO_DOWNLOAD_MANY
async def downloader_coro(cc_list, base_url, verbose, concur_req): # <1>
counter = collections.Counter()
semaphore = asyncio.Semaphore(concur_req) # <2>
to_do = [download_one(cc, base_url, semaphore, verbose)
for cc in sorted(cc_list)] # <3>
to_do_iter = asyncio.as_completed(to_do) # <4>
if not verbose:
to_do_iter = tqdm.tqdm(to_do_iter, total=len(cc_list)) # <5>
for future in to_do_iter: # <6>
try:
res = await future # <7>
except FetchError as exc: # <8>
country_code = exc.country_code # <9>
try:
error_msg = exc.__cause__.args[0] # <10>
except IndexError:
error_msg = exc.__cause__.__class__.__name__ # <11>
if verbose and error_msg:
msg = '*** Error for {}: {}'
print(msg.format(country_code, error_msg))
status = HTTPStatus.error
else:
status = res.status
counter[status] += 1 # <12>
return counter # <13>
def download_many(cc_list, base_url, verbose, concur_req):
loop = asyncio.get_event_loop()
coro = downloader_coro(cc_list, base_url, verbose, concur_req)
counts = loop.run_until_complete(coro) # <14>
loop.close() # <15>
return counts
if __name__ == '__main__':
main(download_many, DEFAULT_CONCUR_REQ, MAX_CONCUR_REQ)
# END FLAGS2_ASYNCIO_DOWNLOAD_MANY

149
20-futures/countries/flags2_common.py
View File

@@ -1,149 +0,0 @@
"""Utilities for second set of flag examples.
"""
import os
import time
import sys
import string
import argparse
from collections import namedtuple
from enum import Enum
Result = namedtuple('Result', 'status data')
HTTPStatus = Enum('Status', 'ok not_found error')
POP20_CC = ('CN IN US ID BR PK NG BD RU JP '
'MX PH VN ET EG DE IR TR CD FR').split()
DEFAULT_CONCUR_REQ = 1
MAX_CONCUR_REQ = 1
SERVERS = {
'REMOTE': 'http://flupy.org/data/flags',
'LOCAL': 'http://localhost:8001/flags',
'DELAY': 'http://localhost:8002/flags',
'ERROR': 'http://localhost:8003/flags',
}
DEFAULT_SERVER = 'LOCAL'
DEST_DIR = 'downloads/'
COUNTRY_CODES_FILE = 'country_codes.txt'
def save_flag(img, filename):
path = os.path.join(DEST_DIR, filename)
with open(path, 'wb') as fp:
fp.write(img)
def initial_report(cc_list, actual_req, server_label):
if len(cc_list) <= 10:
cc_msg = ', '.join(cc_list)
else:
cc_msg = 'from {} to {}'.format(cc_list[0], cc_list[-1])
print('{} site: {}'.format(server_label, SERVERS[server_label]))
msg = 'Searching for {} flag{}: {}'
plural = 's' if len(cc_list) != 1 else ''
print(msg.format(len(cc_list), plural, cc_msg))
plural = 's' if actual_req != 1 else ''
msg = '{} concurrent connection{} will be used.'
print(msg.format(actual_req, plural))
def final_report(cc_list, counter, start_time):
elapsed = time.time() - start_time
print('-' * 20)
msg = '{} flag{} downloaded.'
plural = 's' if counter[HTTPStatus.ok] != 1 else ''
print(msg.format(counter[HTTPStatus.ok], plural))
if counter[HTTPStatus.not_found]:
print(counter[HTTPStatus.not_found], 'not found.')
if counter[HTTPStatus.error]:
plural = 's' if counter[HTTPStatus.error] != 1 else ''
print('{} error{}.'.format(counter[HTTPStatus.error], plural))
print('Elapsed time: {:.2f}s'.format(elapsed))
def expand_cc_args(every_cc, all_cc, cc_args, limit):
codes = set()
A_Z = string.ascii_uppercase
if every_cc:
codes.update(a+b for a in A_Z for b in A_Z)
elif all_cc:
with open(COUNTRY_CODES_FILE) as fp:
text = fp.read()
codes.update(text.split())
else:
for cc in (c.upper() for c in cc_args):
if len(cc) == 1 and cc in A_Z:
codes.update(cc+c for c in A_Z)
elif len(cc) == 2 and all(c in A_Z for c in cc):
codes.add(cc)
else:
msg = 'each CC argument must be A to Z or AA to ZZ.'
raise ValueError('*** Usage error: '+msg)
return sorted(codes)[:limit]
def process_args(default_concur_req):
server_options = ', '.join(sorted(SERVERS))
parser = argparse.ArgumentParser(
description='Download flags for country codes. '
'Default: top 20 countries by population.')
parser.add_argument('cc', metavar='CC', nargs='*',
help='country code or 1st letter (eg. B for BA...BZ)')
parser.add_argument('-a', '--all', action='store_true',
help='get all available flags (AD to ZW)')
parser.add_argument('-e', '--every', action='store_true',
help='get flags for every possible code (AA...ZZ)')
parser.add_argument('-l', '--limit', metavar='N', type=int,
help='limit to N first codes', default=sys.maxsize)
parser.add_argument('-m', '--max_req', metavar='CONCURRENT', type=int,
default=default_concur_req,
help='maximum concurrent requests (default={})'
.format(default_concur_req))
parser.add_argument('-s', '--server', metavar='LABEL',
default=DEFAULT_SERVER,
help='Server to hit; one of {} (default={})'
.format(server_options, DEFAULT_SERVER))
parser.add_argument('-v', '--verbose', action='store_true',
help='output detailed progress info')
args = parser.parse_args()
if args.max_req < 1:
print('*** Usage error: --max_req CONCURRENT must be >= 1')
parser.print_usage()
sys.exit(1)
if args.limit < 1:
print('*** Usage error: --limit N must be >= 1')
parser.print_usage()
sys.exit(1)
args.server = args.server.upper()
if args.server not in SERVERS:
print('*** Usage error: --server LABEL must be one of',
server_options)
parser.print_usage()
sys.exit(1)
try:
cc_list = expand_cc_args(args.every, args.all, args.cc, args.limit)
except ValueError as exc:
print(exc.args[0])
parser.print_usage()
sys.exit(1)
if not cc_list:
cc_list = sorted(POP20_CC)
return args, cc_list
def main(download_many, default_concur_req, max_concur_req):
args, cc_list = process_args(default_concur_req)
actual_req = min(args.max_req, max_concur_req, len(cc_list))
initial_report(cc_list, actual_req, args.server)
base_url = SERVERS[args.server]
t0 = time.time()
counter = download_many(cc_list, base_url, args.verbose, actual_req)
assert sum(counter.values()) == len(cc_list), \
'some downloads are unaccounted for'
final_report(cc_list, counter, t0)

87
20-futures/countries/flags2_sequential.py
View File

@@ -1,87 +0,0 @@
"""Download flags of countries (with error handling).
Sequential version
Sample run::
$ python3 flags2_sequential.py -s DELAY b
DELAY site: http://localhost:8002/flags
Searching for 26 flags: from BA to BZ
1 concurrent connection will be used.
--------------------
17 flags downloaded.
9 not found.
Elapsed time: 13.36s
"""
import collections
import requests
import tqdm
from flags2_common import main, save_flag, HTTPStatus, Result
DEFAULT_CONCUR_REQ = 1
MAX_CONCUR_REQ = 1
# BEGIN FLAGS2_BASIC_HTTP_FUNCTIONS
def get_flag(base_url, cc):
url = '{}/{cc}/{cc}.gif'.format(base_url, cc=cc.lower())
resp = requests.get(url)
if resp.status_code != 200: # <1>
resp.raise_for_status()
return resp.content
def download_one(cc, base_url, verbose=False):
try:
image = get_flag(base_url, cc)
except requests.exceptions.HTTPError as exc: # <2>
res = exc.response
if res.status_code == 404:
status = HTTPStatus.not_found # <3>
msg = 'not found'
else: # <4>
raise
else:
save_flag(image, cc.lower() + '.gif')
status = HTTPStatus.ok
msg = 'OK'
if verbose: # <5>
print(cc, msg)
return Result(status, cc) # <6>
# END FLAGS2_BASIC_HTTP_FUNCTIONS
# BEGIN FLAGS2_DOWNLOAD_MANY_SEQUENTIAL
def download_many(cc_list, base_url, verbose, max_req):
counter = collections.Counter() # <1>
cc_iter = sorted(cc_list) # <2>
if not verbose:
cc_iter = tqdm.tqdm(cc_iter) # <3>
for cc in cc_iter: # <4>
try:
res = download_one(cc, base_url, verbose) # <5>
except requests.exceptions.HTTPError as exc: # <6>
error_msg = 'HTTP error {res.status_code} - {res.reason}'
error_msg = error_msg.format(res=exc.response)
except requests.exceptions.ConnectionError as exc: # <7>
error_msg = 'Connection error'
else: # <8>
error_msg = ''
status = res.status
if error_msg:
status = HTTPStatus.error # <9>
counter[status] += 1 # <10>
if verbose and error_msg: # <11>
print('*** Error for {}: {}'.format(cc, error_msg))
return counter # <12>
# END FLAGS2_DOWNLOAD_MANY_SEQUENTIAL
if __name__ == '__main__':
main(download_many, DEFAULT_CONCUR_REQ, MAX_CONCUR_REQ)

68
20-futures/countries/flags2_threadpool.py
View File

@@ -1,68 +0,0 @@
"""Download flags of countries (with error handling).
ThreadPool version
Sample run::
$ python3 flags2_threadpool.py -s ERROR -e
ERROR site: http://localhost:8003/flags
Searching for 676 flags: from AA to ZZ
30 concurrent connections will be used.
--------------------
150 flags downloaded.
361 not found.
165 errors.
Elapsed time: 7.46s
"""
# BEGIN FLAGS2_THREADPOOL
import collections
from concurrent import futures
import requests
import tqdm # <1>
from flags2_common import main, HTTPStatus # <2>
from flags2_sequential import download_one # <3>
DEFAULT_CONCUR_REQ = 30 # <4>
MAX_CONCUR_REQ = 1000 # <5>
def download_many(cc_list, base_url, verbose, concur_req):
counter = collections.Counter()
with futures.ThreadPoolExecutor(max_workers=concur_req) as executor: # <6>
to_do_map = {} # <7>
for cc in sorted(cc_list): # <8>
future = executor.submit(download_one,
cc, base_url, verbose) # <9>
to_do_map[future] = cc # <10>
done_iter = futures.as_completed(to_do_map) # <11>
if not verbose:
done_iter = tqdm.tqdm(done_iter, total=len(cc_list)) # <12>
for future in done_iter: # <13>
try:
res = future.result() # <14>
except requests.exceptions.HTTPError as exc: # <15>
error_msg = 'HTTP {res.status_code} - {res.reason}'
error_msg = error_msg.format(res=exc.response)
except requests.exceptions.ConnectionError as exc:
error_msg = 'Connection error'
else:
error_msg = ''
status = res.status
if error_msg:
status = HTTPStatus.error
counter[status] += 1
if verbose and error_msg:
cc = to_do_map[future] # <16>
print('*** Error for {}: {}'.format(cc, error_msg))
return counter
if __name__ == '__main__':
main(download_many, DEFAULT_CONCUR_REQ, MAX_CONCUR_REQ)
# END FLAGS2_THREADPOOL

125
20-futures/countries/flags3_asyncio.py
View File

@@ -1,125 +0,0 @@
"""Download flags and names of countries.
asyncio version
"""
import asyncio
import collections
import aiohttp
from aiohttp import web
import tqdm
from flags2_common import main, HTTPStatus, Result, save_flag
# default set low to avoid errors from remote site, such as
# 503 - Service Temporarily Unavailable
DEFAULT_CONCUR_REQ = 5
MAX_CONCUR_REQ = 1000
class FetchError(Exception):
def __init__(self, country_code):
self.country_code = country_code
# BEGIN FLAGS3_ASYNCIO
@asyncio.coroutine
def http_get(url):
res = yield from aiohttp.request('GET', url)
if res.status == 200:
ctype = res.headers.get('Content-type', '').lower()
if 'json' in ctype or url.endswith('json'):
data = yield from res.json() # <1>
else:
data = yield from res.read() # <2>
return data
elif res.status == 404:
raise web.HTTPNotFound()
else:
raise aiohttp.errors.HttpProcessingError(
code=res.status, message=res.reason,
headers=res.headers)
@asyncio.coroutine
def get_country(base_url, cc):
url = '{}/{cc}/metadata.json'.format(base_url, cc=cc.lower())
metadata = yield from http_get(url) # <3>
return metadata['country']
@asyncio.coroutine
def get_flag(base_url, cc):
url = '{}/{cc}/{cc}.gif'.format(base_url, cc=cc.lower())
return (yield from http_get(url)) # <4>
@asyncio.coroutine
def download_one(cc, base_url, semaphore, verbose):
try:
with (yield from semaphore): # <5>
image = yield from get_flag(base_url, cc)
with (yield from semaphore):
country = yield from get_country(base_url, cc)
except web.HTTPNotFound:
status = HTTPStatus.not_found
msg = 'not found'
except Exception as exc:
raise FetchError(cc) from exc
else:
country = country.replace(' ', '_')
filename = '{}-{}.gif'.format(country, cc)
loop = asyncio.get_event_loop()
loop.run_in_executor(None, save_flag, image, filename)
status = HTTPStatus.ok
msg = 'OK'
if verbose and msg:
print(cc, msg)
return Result(status, cc)
# END FLAGS3_ASYNCIO
@asyncio.coroutine
def downloader_coro(cc_list, base_url, verbose, concur_req):
counter = collections.Counter()
semaphore = asyncio.Semaphore(concur_req)
to_do = [download_one(cc, base_url, semaphore, verbose)
for cc in sorted(cc_list)]
to_do_iter = asyncio.as_completed(to_do)
if not verbose:
to_do_iter = tqdm.tqdm(to_do_iter, total=len(cc_list))
for future in to_do_iter:
try:
res = yield from future
except FetchError as exc:
country_code = exc.country_code
try:
error_msg = exc.__cause__.args[0]
except IndexError:
error_msg = exc.__cause__.__class__.__name__
if verbose and error_msg:
msg = '*** Error for {}: {}'
print(msg.format(country_code, error_msg))
status = HTTPStatus.error
else:
status = res.status
counter[status] += 1
return counter
def download_many(cc_list, base_url, verbose, concur_req):
loop = asyncio.get_event_loop()
coro = downloader_coro(cc_list, base_url, verbose, concur_req)
counts = loop.run_until_complete(coro)
loop.close()
return counts
if __name__ == '__main__':
main(download_many, DEFAULT_CONCUR_REQ, MAX_CONCUR_REQ)

84
20-futures/countries/flags3_threadpool.py
View File

@@ -1,84 +0,0 @@
"""Download flags and names of countries.
ThreadPool version
"""
import collections
from concurrent import futures
import requests
import tqdm
from flags2_common import main, save_flag, HTTPStatus, Result
from flags2_sequential import get_flag
DEFAULT_CONCUR_REQ = 30
MAX_CONCUR_REQ = 1000
def get_country(base_url, cc):
url = '{}/{cc}/metadata.json'.format(base_url, cc=cc.lower())
res = requests.get(url)
if res.status_code != 200:
res.raise_for_status()
return res.json()['country']
def download_one(cc, base_url, verbose=False):
try:
image = get_flag(base_url, cc)
country = get_country(base_url, cc)
except requests.exceptions.HTTPError as exc:
res = exc.response
if res.status_code == 404:
status = HTTPStatus.not_found
msg = 'not found'
else: # <4>
raise
else:
country = country.replace(' ', '_')
save_flag(image, '{}-{}.gif'.format(country, cc))
status = HTTPStatus.ok
msg = 'OK'
if verbose:
print(cc, msg)
return Result(status, cc)
def download_many(cc_list, base_url, verbose, concur_req):
counter = collections.Counter()
with futures.ThreadPoolExecutor(concur_req) as executor:
to_do_map = {}
for cc in sorted(cc_list):
future = executor.submit(download_one,
cc, base_url, verbose)
to_do_map[future] = cc
to_do_iter = futures.as_completed(to_do_map)
if not verbose:
to_do_iter = tqdm.tqdm(to_do_iter, total=len(cc_list))
for future in to_do_iter:
try:
res = future.result()
except requests.exceptions.HTTPError as exc:
error_msg = 'HTTP {res.status_code} - {res.reason}'
error_msg = error_msg.format(res=exc.response)
except requests.exceptions.ConnectionError as exc:
error_msg = 'Connection error'
else:
error_msg = ''
status = res.status
if error_msg:
status = HTTPStatus.error
counter[status] += 1
if verbose and error_msg:
cc = to_do_map[future]
print('*** Error for {}: {}'.format(cc, error_msg))
return counter
if __name__ == '__main__':
main(download_many, DEFAULT_CONCUR_REQ, MAX_CONCUR_REQ)

48
20-futures/countries/flags_asyncio.py
View File

@@ -1,48 +0,0 @@
"""Download flags of top 20 countries by population
asyncio + aiottp version
Sample run::
$ python3 flags_asyncio.py
EG VN IN TR RU ID US DE CN MX JP BD NG ET FR BR PH PK CD IR
20 flags downloaded in 1.07s
"""
# BEGIN FLAGS_ASYNCIO
import asyncio
import aiohttp # <1>
from flags import BASE_URL, save_flag, show, main # <2>
@asyncio.coroutine # <3>
def get_flag(cc):
url = '{}/{cc}/{cc}.gif'.format(BASE_URL, cc=cc.lower())
resp = yield from aiohttp.request('GET', url) # <4>
image = yield from resp.read() # <5>
return image
@asyncio.coroutine
def download_one(cc): # <6>
image = yield from get_flag(cc) # <7>
show(cc)
save_flag(image, cc.lower() + '.gif')
return cc
def download_many(cc_list):
loop = asyncio.get_event_loop() # <8>
to_do = [download_one(cc) for cc in sorted(cc_list)] # <9>
wait_coro = asyncio.wait(to_do) # <10>
res, _ = loop.run_until_complete(wait_coro) # <11>
loop.close() # <12>
return len(res)
if __name__ == '__main__':
main(download_many)
# END FLAGS_ASYNCIO

46
20-futures/countries/flags_await.py
View File

@@ -1,46 +0,0 @@
"""Download flags of top 20 countries by population
asyncio + aiottp version
Sample run::
$ python3 flags_asyncio.py
EG VN IN TR RU ID US DE CN MX JP BD NG ET FR BR PH PK CD IR
20 flags downloaded in 1.07s
"""
# BEGIN FLAGS_ASYNCIO
import asyncio
import aiohttp # <1>
from flags import BASE_URL, save_flag, show, main # <2>
async def get_flag(cc): # <3>
url = '{}/{cc}/{cc}.gif'.format(BASE_URL, cc=cc.lower())
resp = await aiohttp.request('GET', url) # <4>
image = await resp.read() # <5>
return image
async def download_one(cc): # <6>
image = await get_flag(cc) # <7>
show(cc)
save_flag(image, cc.lower() + '.gif')
return cc
def download_many(cc_list):
loop = asyncio.get_event_loop() # <8>
to_do = [download_one(cc) for cc in sorted(cc_list)] # <9>
wait_coro = asyncio.wait(to_do) # <10>
res, _ = loop.run_until_complete(wait_coro) # <11>
loop.close() # <12>
return len(res)
if __name__ == '__main__':
main(download_many)
# END FLAGS_ASYNCIO

43
20-futures/countries/flags_threadpool.py
View File

@@ -1,43 +0,0 @@
"""Download flags of top 20 countries by population
ThreadPoolExecutor version
Sample run::
$ python3 flags_threadpool.py
BD retrieved.
EG retrieved.
CN retrieved.
...
PH retrieved.
US retrieved.
IR retrieved.
20 flags downloaded in 0.93s
"""
# BEGIN FLAGS_THREADPOOL
from concurrent import futures
from flags import save_flag, get_flag, show, main # <1>
MAX_WORKERS = 20 # <2>
def download_one(cc): # <3>
image = get_flag(cc)
show(cc)
save_flag(image, cc.lower() + '.gif')
return cc
def download_many(cc_list):
workers = min(MAX_WORKERS, len(cc_list)) # <4>
with futures.ThreadPoolExecutor(workers) as executor: # <5>
res = executor.map(download_one, sorted(cc_list)) # <6>
return len(list(res)) # <7>
if __name__ == '__main__':
main(download_many) # <8>
# END FLAGS_THREADPOOL

55
20-futures/countries/flags_threadpool_ac.py
View File

@@ -1,55 +0,0 @@
"""Download flags of top 20 countries by population
ThreadPoolExecutor version 2, with ``as_completed``.
Sample run::
$ python3 flags_threadpool.py
BD retrieved.
EG retrieved.
CN retrieved.
...
PH retrieved.
US retrieved.
IR retrieved.
20 flags downloaded in 0.93s
"""
from concurrent import futures
from flags import save_flag, get_flag, show, main
MAX_WORKERS = 20
def download_one(cc):
image = get_flag(cc)
show(cc)
save_flag(image, cc.lower() + '.gif')
return cc
# BEGIN FLAGS_THREADPOOL_AS_COMPLETED
def download_many(cc_list):
cc_list = cc_list[:5] # <1>
with futures.ThreadPoolExecutor(max_workers=3) as executor: # <2>
to_do = []
for cc in sorted(cc_list): # <3>
future = executor.submit(download_one, cc) # <4>
to_do.append(future) # <5>
msg = 'Scheduled for {}: {}'
print(msg.format(cc, future)) # <6>
results = []
for future in futures.as_completed(to_do): # <7>
res = future.result() # <8>
msg = '{} result: {!r}'
print(msg.format(future, res)) # <9>
results.append(res)
return len(results)
# END FLAGS_THREADPOOL_AS_COMPLETED
if __name__ == '__main__':
main(download_many)

3
20-futures/countries/requirements.txt
View File

@@ -1,3 +0,0 @@
aiohttp==0.13.1
requests==2.21.0
tqdm==1.0

4
20-futures/countries/vaurien_delay.sh
View File

@@ -1,4 +0,0 @@
#!/bin/bash
vaurien --protocol http --backend localhost:8001 \
--proxy localhost:8002 \
--behavior 100:delay --behavior-delay-sleep .5

4
20-futures/countries/vaurien_error_delay.sh
View File

@@ -1,4 +0,0 @@
#!/bin/bash
vaurien --protocol http --backend localhost:8001 \
--proxy localhost:8003 \
--behavior 25:error,50:delay --behavior-delay-sleep .5

65
20-futures/crypto/arcfour-timings.txt
View File

@@ -1,65 +0,0 @@
workers|time
4|4.96
3|5.40
2|8.35
1|11.25
1|11.17
2|8.45
3|6.08
4|5.83
4|6.22
3|7.33
2|9.48
1|11.86
1|11.72
2|9.22
3|6.74
4|6.37
4|4.94
3|5.51
2|8.25
1|11.47
1|12.90
2|8.94
3|6.44
4|5.90
4|5.94
3|6.46
2|9.10
1|11.66
1|11.48
2|9.08
3|6.31
4|5.99
4|5.02
3|5.46
2|8.26
1|11.18
1|11.23
2|8.52
3|5.64
4|5.39
4|5.53
3|6.07
2|8.66
1|11.42
1|11.34
2|8.44
3|5.88
4|5.57
4|4.93
3|5.47
2|8.65
1|11.23
1|11.12
2|7.83
3|5.81
4|5.45
4|5.54
3|6.09
2|8.84
1|11.45
1|11.25
2|8.32
3|6.02
4|5.74

55
20-futures/crypto/arcfour.py
View File

@@ -1,55 +0,0 @@
"""RC4 compatible algorithm"""
def arcfour(key, in_bytes, loops=20):
kbox = bytearray(256) # create key box
for i, car in enumerate(key): # copy key and vector
kbox[i] = car
j = len(key)
for i in range(j, 256): # repeat until full
kbox[i] = kbox[i-j]
# [1] initialize sbox
sbox = bytearray(range(256))
# repeat sbox mixing loop, as recommened in CipherSaber-2
# http://ciphersaber.gurus.com/faq.html#cs2
j = 0
for k in range(loops):
for i in range(256):
j = (j + sbox[i] + kbox[i]) % 256
sbox[i], sbox[j] = sbox[j], sbox[i]
# main loop
i = 0
j = 0
out_bytes = bytearray()
for car in in_bytes:
i = (i + 1) % 256
# [2] shuffle sbox
j = (j + sbox[i]) % 256
sbox[i], sbox[j] = sbox[j], sbox[i]
# [3] compute t
t = (sbox[i] + sbox[j]) % 256
k = sbox[t]
car = car ^ k
out_bytes.append(car)
return out_bytes
def test():
from time import time
clear = bytearray(b'1234567890' * 100000)
t0 = time()
cipher = arcfour(b'key', clear)
print('elapsed time: %.2fs' % (time() - t0))
result = arcfour(b'key', cipher)
assert result == clear, '%r != %r' % (result, clear)
print('elapsed time: %.2fs' % (time() - t0))
print('OK')
if __name__ == '__main__':
test()

46
20-futures/crypto/arcfour_futures.py
View File

@@ -1,46 +0,0 @@
import sys
import time
from concurrent import futures
from random import randrange
from arcfour import arcfour
JOBS = 12
SIZE = 2**18
KEY = b"'Twas brillig, and the slithy toves\nDid gyre"
STATUS = '{} workers, elapsed time: {:.2f}s'
def arcfour_test(size, key):
in_text = bytearray(randrange(256) for i in range(size))
cypher_text = arcfour(key, in_text)
out_text = arcfour(key, cypher_text)
assert in_text == out_text, 'Failed arcfour_test'
return size
def main(workers=None):
if workers:
workers = int(workers)
t0 = time.time()
with futures.ProcessPoolExecutor(workers) as executor:
actual_workers = executor._max_workers
to_do = []
for i in range(JOBS, 0, -1):
size = SIZE + int(SIZE / JOBS * (i - JOBS/2))
job = executor.submit(arcfour_test, size, KEY)
to_do.append(job)
for future in futures.as_completed(to_do):
res = future.result()
print('{:.1f} KB'.format(res/2**10))
print(STATUS.format(actual_workers, time.time() - t0))
if __name__ == '__main__':
if len(sys.argv) == 2:
workers = int(sys.argv[1])
else:
workers = None
main(workers)

119
20-futures/crypto/arcfour_test.py
View File

@@ -1,119 +0,0 @@
#!/usr/bin/env python3
from arcfour import arcfour
'''
Source of the test vectors:
A Stream Cipher Encryption Algorithm "Arcfour"
http://tools.ietf.org/html/draft-kaukonen-cipher-arcfour-03
'''
TEST_VECTORS = [
('CRYPTLIB', {
'Plain Text' : (0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00),
'Key' : (0x01, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD, 0xEF),
'Cipher Text' : (0x74, 0x94, 0xC2, 0xE7, 0x10, 0x4B, 0x08, 0x79),
}
),
('COMMERCE', {
'Plain Text' : (0xdc, 0xee, 0x4c, 0xf9, 0x2c),
'Key' : (0x61, 0x8a, 0x63, 0xd2, 0xfb),
'Cipher Text' : (0xf1, 0x38, 0x29, 0xc9, 0xde),
}
),
('SSH ARCFOUR', {
'Plain Text' : (
0x52, 0x75, 0x69, 0x73, 0x6c, 0x69, 0x6e, 0x6e,
0x75, 0x6e, 0x20, 0x6c, 0x61, 0x75, 0x6c, 0x75,
0x20, 0x6b, 0x6f, 0x72, 0x76, 0x69, 0x73, 0x73,
0x73, 0x61, 0x6e, 0x69, 0x2c, 0x20, 0x74, 0xe4,
0x68, 0x6b, 0xe4, 0x70, 0xe4, 0x69, 0x64, 0x65,
0x6e, 0x20, 0x70, 0xe4, 0xe4, 0x6c, 0x6c, 0xe4,
0x20, 0x74, 0xe4, 0x79, 0x73, 0x69, 0x6b, 0x75,
0x75, 0x2e, 0x20, 0x4b, 0x65, 0x73, 0xe4, 0x79,
0xf6, 0x6e, 0x20, 0x6f, 0x6e, 0x20, 0x6f, 0x6e,
0x6e, 0x69, 0x20, 0x6f, 0x6d, 0x61, 0x6e, 0x61,
0x6e, 0x69, 0x2c, 0x20, 0x6b, 0x61, 0x73, 0x6b,
0x69, 0x73, 0x61, 0x76, 0x75, 0x75, 0x6e, 0x20,
0x6c, 0x61, 0x61, 0x6b, 0x73, 0x6f, 0x74, 0x20,
0x76, 0x65, 0x72, 0x68, 0x6f, 0x75, 0x75, 0x2e,
0x20, 0x45, 0x6e, 0x20, 0x6d, 0x61, 0x20, 0x69,
0x6c, 0x6f, 0x69, 0x74, 0x73, 0x65, 0x2c, 0x20,
0x73, 0x75, 0x72, 0x65, 0x20, 0x68, 0x75, 0x6f,
0x6b, 0x61, 0x61, 0x2c, 0x20, 0x6d, 0x75, 0x74,
0x74, 0x61, 0x20, 0x6d, 0x65, 0x74, 0x73, 0xe4,
0x6e, 0x20, 0x74, 0x75, 0x6d, 0x6d, 0x75, 0x75,
0x73, 0x20, 0x6d, 0x75, 0x6c, 0x6c, 0x65, 0x20,
0x74, 0x75, 0x6f, 0x6b, 0x61, 0x61, 0x2e, 0x20,
0x50, 0x75, 0x75, 0x6e, 0x74, 0x6f, 0x20, 0x70,
0x69, 0x6c, 0x76, 0x65, 0x6e, 0x2c, 0x20, 0x6d,
0x69, 0x20, 0x68, 0x75, 0x6b, 0x6b, 0x75, 0x75,
0x2c, 0x20, 0x73, 0x69, 0x69, 0x6e, 0x74, 0x6f,
0x20, 0x76, 0x61, 0x72, 0x61, 0x6e, 0x20, 0x74,
0x75, 0x75, 0x6c, 0x69, 0x73, 0x65, 0x6e, 0x2c,
0x20, 0x6d, 0x69, 0x20, 0x6e, 0x75, 0x6b, 0x6b,
0x75, 0x75, 0x2e, 0x20, 0x54, 0x75, 0x6f, 0x6b,
0x73, 0x75, 0x74, 0x20, 0x76, 0x61, 0x6e, 0x61,
0x6d, 0x6f, 0x6e, 0x20, 0x6a, 0x61, 0x20, 0x76,
0x61, 0x72, 0x6a, 0x6f, 0x74, 0x20, 0x76, 0x65,
0x65, 0x6e, 0x2c, 0x20, 0x6e, 0x69, 0x69, 0x73,
0x74, 0xe4, 0x20, 0x73, 0x79, 0x64, 0xe4, 0x6d,
0x65, 0x6e, 0x69, 0x20, 0x6c, 0x61, 0x75, 0x6c,
0x75, 0x6e, 0x20, 0x74, 0x65, 0x65, 0x6e, 0x2e,
0x20, 0x2d, 0x20, 0x45, 0x69, 0x6e, 0x6f, 0x20,
0x4c, 0x65, 0x69, 0x6e, 0x6f),
'Key' : (
0x29, 0x04, 0x19, 0x72, 0xfb, 0x42, 0xba, 0x5f,
0xc7, 0x12, 0x77, 0x12, 0xf1, 0x38, 0x29, 0xc9),
'Cipher Text' : (
0x35, 0x81, 0x86, 0x99, 0x90, 0x01, 0xe6, 0xb5,
0xda, 0xf0, 0x5e, 0xce, 0xeb, 0x7e, 0xee, 0x21,
0xe0, 0x68, 0x9c, 0x1f, 0x00, 0xee, 0xa8, 0x1f,
0x7d, 0xd2, 0xca, 0xae, 0xe1, 0xd2, 0x76, 0x3e,
0x68, 0xaf, 0x0e, 0xad, 0x33, 0xd6, 0x6c, 0x26,
0x8b, 0xc9, 0x46, 0xc4, 0x84, 0xfb, 0xe9, 0x4c,
0x5f, 0x5e, 0x0b, 0x86, 0xa5, 0x92, 0x79, 0xe4,
0xf8, 0x24, 0xe7, 0xa6, 0x40, 0xbd, 0x22, 0x32,
0x10, 0xb0, 0xa6, 0x11, 0x60, 0xb7, 0xbc, 0xe9,
0x86, 0xea, 0x65, 0x68, 0x80, 0x03, 0x59, 0x6b,
0x63, 0x0a, 0x6b, 0x90, 0xf8, 0xe0, 0xca, 0xf6,
0x91, 0x2a, 0x98, 0xeb, 0x87, 0x21, 0x76, 0xe8,
0x3c, 0x20, 0x2c, 0xaa, 0x64, 0x16, 0x6d, 0x2c,
0xce, 0x57, 0xff, 0x1b, 0xca, 0x57, 0xb2, 0x13,
0xf0, 0xed, 0x1a, 0xa7, 0x2f, 0xb8, 0xea, 0x52,
0xb0, 0xbe, 0x01, 0xcd, 0x1e, 0x41, 0x28, 0x67,
0x72, 0x0b, 0x32, 0x6e, 0xb3, 0x89, 0xd0, 0x11,
0xbd, 0x70, 0xd8, 0xaf, 0x03, 0x5f, 0xb0, 0xd8,
0x58, 0x9d, 0xbc, 0xe3, 0xc6, 0x66, 0xf5, 0xea,
0x8d, 0x4c, 0x79, 0x54, 0xc5, 0x0c, 0x3f, 0x34,
0x0b, 0x04, 0x67, 0xf8, 0x1b, 0x42, 0x59, 0x61,
0xc1, 0x18, 0x43, 0x07, 0x4d, 0xf6, 0x20, 0xf2,
0x08, 0x40, 0x4b, 0x39, 0x4c, 0xf9, 0xd3, 0x7f,
0xf5, 0x4b, 0x5f, 0x1a, 0xd8, 0xf6, 0xea, 0x7d,
0xa3, 0xc5, 0x61, 0xdf, 0xa7, 0x28, 0x1f, 0x96,
0x44, 0x63, 0xd2, 0xcc, 0x35, 0xa4, 0xd1, 0xb0,
0x34, 0x90, 0xde, 0xc5, 0x1b, 0x07, 0x11, 0xfb,
0xd6, 0xf5, 0x5f, 0x79, 0x23, 0x4d, 0x5b, 0x7c,
0x76, 0x66, 0x22, 0xa6, 0x6d, 0xe9, 0x2b, 0xe9,
0x96, 0x46, 0x1d, 0x5e, 0x4d, 0xc8, 0x78, 0xef,
0x9b, 0xca, 0x03, 0x05, 0x21, 0xe8, 0x35, 0x1e,
0x4b, 0xae, 0xd2, 0xfd, 0x04, 0xf9, 0x46, 0x73,
0x68, 0xc4, 0xad, 0x6a, 0xc1, 0x86, 0xd0, 0x82,
0x45, 0xb2, 0x63, 0xa2, 0x66, 0x6d, 0x1f, 0x6c,
0x54, 0x20, 0xf1, 0x59, 0x9d, 0xfd, 0x9f, 0x43,
0x89, 0x21, 0xc2, 0xf5, 0xa4, 0x63, 0x93, 0x8c,
0xe0, 0x98, 0x22, 0x65, 0xee, 0xf7, 0x01, 0x79,
0xbc, 0x55, 0x3f, 0x33, 0x9e, 0xb1, 0xa4, 0xc1,
0xaf, 0x5f, 0x6a, 0x54, 0x7f),
}
),
]
for name, vectors in TEST_VECTORS:
print(name, end='')
plain = bytearray(vectors['Plain Text'])
cipher = bytearray(vectors['Cipher Text'])
key = bytearray(vectors['Key'])
assert cipher == arcfour(key, plain, loops=1)
assert plain == arcfour(key, cipher, loops=1)
print(' --> OK')

50
20-futures/crypto/sha-timings.txt
View File

@@ -1,50 +0,0 @@
workers|time
4|8.88
3|11.14
2|13.66
1|22.80
1|25.42
2|16.37
3|12.09
4|11.06
4|11.40
3|11.51
2|15.20
1|24.18
1|22.09
2|12.48
3|10.78
4|10.48
4|8.48
3|10.07
2|12.42
1|20.24
1|20.31
2|11.39
3|10.88
4|10.44
4|10.43
3|11.11
2|12.39
1|20.69
1|20.53
2|11.80
3|11.01
4|10.52
4|11.50
3|14.45
2|16.95
1|24.77
1|22.71
2|18.35
3|12.66
4|12.20
4|12.37
3|13.37
2|19.30
1|24.30
1|23.93
2|18.51
3|13.88
4|12.97

38
20-futures/crypto/sha_futures.py
View File

@@ -1,38 +0,0 @@
import sys
import time
import hashlib
from concurrent import futures
from random import randrange
JOBS = 12
SIZE = 2**20
STATUS = '{} workers, elapsed time: {:.2f}s'
def sha(size):
data = bytearray(randrange(256) for i in range(size))
algo = hashlib.new('sha256')
algo.update(data)
return algo.hexdigest()
def main(workers=None):
if workers:
workers = int(workers)
t0 = time.time()
with futures.ProcessPoolExecutor(workers) as executor:
actual_workers = executor._max_workers
to_do = (executor.submit(sha, SIZE) for i in range(JOBS))
for future in futures.as_completed(to_do):
res = future.result()
print(res)
print(STATUS.format(actual_workers, time.time() - t0))
if __name__ == '__main__':
if len(sys.argv) == 2:
workers = int(sys.argv[1])
else:
workers = None
main(workers)

34
20-futures/demo_executor_map.py
View File

@@ -1,34 +0,0 @@
"""
Experiment with ``ThreadPoolExecutor.map``
"""
# BEGIN EXECUTOR_MAP
from time import sleep, strftime
from concurrent import futures
def display(*args): # <1>
print(strftime('[%H:%M:%S]'), end=' ')
print(*args)
def loiter(n): # <2>
msg = '{}loiter({}): doing nothing for {}s...'
display(msg.format('\t'*n, n, n))
sleep(n)
msg = '{}loiter({}): done.'
display(msg.format('\t'*n, n))
return n * 10 # <3>
def main():
display('Script starting.')
executor = futures.ThreadPoolExecutor(max_workers=3) # <4>
results = executor.map(loiter, range(5)) # <5>
display('results:', results) # <6>.
display('Waiting for individual results:')
for i, result in enumerate(results): # <7>
display('result {}: {}'.format(i, result))
main()
# END EXECUTOR_MAP