Log file examination

I downloaded one of the access_log files from monitorware.com. Like any other web access log, we have one line per entry, like this:

64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846 

The first part is the IP address of the caller, followed by a timestamp, the type of HTTP access, the URL referenced, the HTTP type, the resulting HTTP response code, and finally the number of bytes in the response.

We can use Spark to load in and parse out some statistics of the log entries, as in this script:

import pyspark
if not 'sc' in globals():
    sc = pyspark.SparkContext()

textFile = sc.textFile("access_log")
print(textFile.count(), "access records")

gets = textFile.filter(lambda line: "GET" in line)
print(gets.count(), "GETs")

posts = textFile.filter(lambda line: "POST" in line)
print(posts.count(), "POSTs")

other = textFile.subtract(gets).subtract(posts)
print(other.count(...