The Social Security administration has this neat data by year of what names are most popular for babies born that year in the USA (see social security baby names).

The files for this exercise are in the "babynames" directory of the downloaded exercises.zip. Add your code in babynames.py. The files baby1990.html baby1992.html ... contain raw html, similar to what you get visiting the above social security site. Take a look at the html and think about how you might scrape the data out of it.

In the babynames.py file, implement the extract_names(filename) function which takes the filename of a baby1990.html file and returns the data from the file as a single list -- the year string at the start of the list followed by the name-rank strings in alphabetical order. ['2006', 'Aaliyah 91', 'Abagail 895', 'Aaron 57', ...]. Modify main() so it calls your extract_names function and prints what it returns (main already has the code for the command line argument parsing). If you get stuck working out the regular expressions for the year and each name, solution regular expression patterns are shown at the end of this document. Note that for parsing webpages in general, regular expressions don't do a good job, but these webpages have a simple and consistent format.

Rather than treat the boy and girl names separately, we'll just lump them all together. In some years, a name appears more than once in the html, but we'll just use one number per name. Optional: make the algorithm smart about this case and choose whichever number is smaller.

Build the program as a series of small milestones, getting each step to run/print something before trying the next step. This is the pattern used by experienced programmers -- build a series of incremental milestones, each with some output to check, rather than building the whole program in one huge step.

Printing the data you have at the end of one milestone helps you think about how to re-structure that data for the next milestone. Python is well suited to this style of incremental development. For example, first get it to the point where it extracts and prints the year and calls sys.exit(0). Here are some suggested milestones:

Earlier we have had functions just print to standard out. It's more re-usable to have the function return the extracted data, so then the caller has the choice to print it or do something else with it. (You can still print directly from inside your functions for your little experiments during development.)

Have main call extract_names for each command line argument and print a text summary. To make the list into a reasonable looking summary text, here's a clever use of join:

text = '\n'.join(mylist) + '\n'

The summary text should look like this for each file:

2006
Aaliyah 91
Aaron 57
Abagail 895
Abbey 695
Abbie 650
...

Suppose instead of printing the text to standard out, we want to write files containing the text. If the flag --summaryfile is present, do the following: for each input file foo.html, create a new file foo.html.summary that contains the summary text for that file.

Once the --summaryfile feature is working, run the program on all the files using * like this:

python babynames.py --summaryfile baby*.html

This generates all the summaries in one step. (The standard behavior of the shell is that it expands the baby*.html pattern into a list of matching filenames, and then the shell runs babynames.py, passing in all those filenames in the sys.argv list.)

With the data organized into summary files, you can see patterns over time with shell commands, like this:

$ grep 'Trinity ' *.summary
$ grep 'Nick ' *.summary
$ grep 'Miguel ' *.summary
$ grep 'Emily ' *.summary

r'Popularity\sin\s(\d\d\d\d)'

r'<td>(\d+)</td><td>(\w+)</td>\<td>(\w+)</td>'

The copyspecial.py program takes one or more directories as its arguments. We'll say that a "special" file is one where the name contains the pattern __w__ somewhere, where the w is one or more word chars. The provided main includes code to parse the command line arguments, but the rest is up to you. Write functions to implement the features below and modify main to call your functions.

Suggested functions for your solution(details below):

Gather a list of the absolute paths of the special files in all the directories. In the simplest case, just print that list (here the . after the command is a single argument indicating the current directory). Print one absolute path per line.

$ python copyspecial.py .
/Users/miller-time/pycourse/day2/xyz__hello__.txt
/Users/miller-time/pycourse/day2/zz__something__.jpg

We'll assume that names are not repeated across the directories (optional: check that assumption and error out if it's violated).

If the --todir dir option is present, do not print anything and instead copy the files to the given directory, creating it if necessary. Use the Python shutil module for file copying.

$ python copyspecial.py --todir /tmp/fooby .
$ ls /tmp/fooby
xyz__hello__.txt        zz__something__.jpg

If the --tozip zipfile option is present, create the zip file using the Python zipfile module.

$ python copyspecial.py --tozip tmp.zip .

An image of an animal has been broken into many narrow vertical stripe images. The stripe images are on the internet somewhere, each with its own URL. The URLs are hidden in a web server log file. Your mission is to find the URLs and download all image stripes to re-create the original image.

The slice URLs are hidden inside apache log files (the open source apache web server is the most widely used server on the internet). Each log file is from some server, and the desired slice URLs are hidden within the logs. The log file encodes what server it comes from like this: the log file animal_code.google.com is from the code.google.com server (formally, we'll say that the server name is whatever follows the first underbar). The animal_code.google.com log file contains the data for the "animal" puzzle image. Although the data in the log files has the syntax of a real apache web server, the data beyond what's needed for the puzzle is randomized data from a real log file.

Here is what a single line from the log file looks like (this really is what apache log files look like):

10.254.254.28 - - [06/Aug/2007:00:14:08 -0700] "GET /foo/talks/ HTTP/1.1"
200 5910 "-" "Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4"

The first few numbers are the address of the requesting browser. The most interesting part is the GET path HTTP showing the path of a web request received by the server. The path itself never contains spaces, and is separated from the GET and HTTP by spaces (regex suggestion: \S (upper case S) matches any non-space char). Find the lines in the log where the string "puzzle" appears inside the path, ignoring the many other lines in the log.

Complete the read_urls function that extracts the puzzle URLs from inside a log file. Find all the "puzzle" path URLs in the log file. Combine the path from each URL with the server name from the filename to form a full URL, e.g. http://www.example.com/path/puzzle/from/inside/file. Exclude URLs that appear more than once. The read_urls function should return the list of full URLs, sorted into alphabetical order and without duplicates. Taking the URLs in alphabetical order will yield the image slices in the correct left-to-right order to re-create the original image. In the simplest case, main should just print the URLs, one per line.

$ python logpuzzle.py animal_code.google.com
http://code.google.com/something/puzzle-animal-baaa.jpg
http://code.google.com/something/puzzle-animal-baab.jpg
...

Complete the download_images function which takes a sorted list of URLs and a directory. Download the image from each URL into the given directory, creating the directory first if necessary (see the os module to create a directory, and urllib.request.urlretrieve for downloading a url). Name the local image files with a simple scheme like "img0", "img1", "img2", and so on. You may wish to print a little "Retrieving..." status output line while downloading each image since it can be slow and it's nice to have some indication that the program is working. Each image is a little vertical slice from the original. How to put the slices together to re-create the original? It can be solved nicely with a little html (knowledge of HTML is not required).

The download_images function should also create an index.html file in the directory with an img tag to show each local image file. The img tags should all be on one line together without separation. In this way, the browser displays all the slices together seamlessly. You do not need knowledge of HTML to do this; just create an index.html file that looks like this:

<html>
<body>
<img src="/edu/python/exercises/img0"><img src="/edu/python/exercises/img1"><img src="/edu/python/exercises/img2">...
</body>
</html>

Here's what it should look like when you download the animal puzzle:

$ python logpuzzle.py --todir animaldir animal_code.google.com
$ ls animaldir
img0  img1  img2  img3  img4  img5  img6  img7  img8  img9  index.html

When it's all working, opening the index.html in a browser should reveal the original animal image. What is the animal in the image?

The second puzzle involves an image of a very famous place but depends on some custom sorting. For the first puzzle, the URLs can be sorted alphabetically to order the images correctly. In the sort, the whole URL is used. However, we'll say that if the URL ends in the pattern -wordchars-wordchars.jpg, e.g. http://example.com/foo/puzzle/bar-abab-baaa.jpg, then the sort should use the second word (e.g. "baaa") for that URL. So sorting a list of URLs each ending with the word-word.jpg pattern should order the URLs by the second word.

Extend your code to order such URLs properly, and then you should be able to decode the place_code.google.com puzzle which shows a famous place. What place does it show?

The official Python documentation for urllib.request recommends using the Requests package for a higher-level HTTP client interface. This is a great opportunity to practice using third party libraries in your scripts.

You can either work in your existing logpuzzle directory or create a new directory for the new version of the puzzle. Within that directory, we want to create a Python virtual environment using this command:

python -m venv .venv

Next we want to activate our new virtual environment:

source .venv/bin/activate

Now we want to create a file called requirements.txt, which will contain a list of the third-party packages that our script depends on.

requests==2.27

This file ensures that our script will always use version 2.27.* of this package. A newer version of the package might be available in the future -- feel free to experiment with newer versions if you'd like.

Next we need to install the package into our active virtual environment.

pip install -r requirements.txt

And now in the logpuzzle.py script, we can import the new package:

import requests

The goal here is to replace our call to urllib.request.urlretrieve. The requests package supports downloading a file from a given URL with the following approach:

r = requests.get(url, stream=True)
with open(filename, 'wb') as fd:
    for chunk in r.iter_content(chunk_size=128):
        fd.write(chunk)

Run your script again to verify that it still works as expected.

$ python logpuzzle.py --todir animaldir animal_code.google.com
$ python logpuzzle.py --todir placedir place_code.google.com

CC Attribution: the images used in this puzzle were made available by their owners under the Creative Commons Attribution 2.5 license, which generously encourages remixes of the content such as this one. The animal image is from the user zappowbang at flickr and the place image is from the user booleansplit at flickr.