Let’s take a look at some throwaway code. Remember, throwaway code is written quickly, with little thought to architecture. It’s good enough to ship but not good enough to survive extension. Consider a program designed to ingest log data from a popular web server, and subsequently analyze and present salient information in the form of a report rendered in HTML. You will be analyzing logs from NGINX (pronounced ‘engine-ex’), one of the most popular web server programs in use today. I usually write a user story in an issue tracker, but I’ll write it as a Markdown file in lieu, and I’ll include it with my project so that I have a record of my requirements:
As an IT administrator, I would like to be able to easily review weblog traffic by running a command that takes in the location on my computer of a log file from a server running NGINX. I would also like to store the data in a relational database table for future analysis.
GIVEN: I have a log file from NGINX on my computer at c:\temp\nginx-sample.log AND
GIVEN: I have opened a PowerShell terminal window in Windows 10 or later AND
GIVEN: The WebLogReporter program is listed within my computer's PATH environment variable.
THEN: I can run the WebLogReporter command, pass the location of the weblog and the path for the output HTML file.
GIVEN: The program runs without errors.
THEN: I am able to view the output HTML file in my favorite browser.
Acceptance Criteria:
* It's done when I can run the WebLogReporter program with no arguments and receive instructions.
* It's done when I can run the WebLogReporter program with two arguments, consisting of the first being a full path to the NGINX log file I wish to analyze and the second being the full path to the output HTML file I would like the program to produce, and I am able to view the output HTML file within my browser.
* It's done when all the log data are stored in a relational database table so I can query and analyze the data later.
Your team decides to use C# and SQL Server to read, parse, and store the data for analysis. They decide that, while there are several good templating systems out there, nobody on the team has ever used any of them. Time is short and HTML is simple, so we’ll just write our own code to convert our results represented by the results of SQL statements. Let’s dive in! The requirements stipulate a console application, so that’s the project type I used when creating it in my IDE. I won’t be walking you through creating the project. I’m assuming you know how to create a console application using the new project options in Visual Studio.
The input data from an NGINX log looks as follows:
127.0.0.1 - - [16/Jan/2022:04:09:51 +0000] "GET /api/get_pricing_info/B641F364-DB29-4241-A45B-7AF6146BC HTTP/1.1" 200 5442 "-" "python-requests/2.25.0"
127.0.0.1 - - [16/Jan/2022:04:09:52 +0000] "GET /api/get_inventory/B641F364-DB29-4241-A45B-7AF6146BC HTTP/1.1" 200 3007 "-" "python-requests/2.25.0"
127.0.0.1 - - [16/Jan/2022:04:09:52 +0000] "GET /api/get_product_details/B641F364-DB29-4241-A45B-7AF6146BC HTTP/1.1" 200 3572 "-" "python-requests/2.25.0"
When you create a console app project in Visual Studio, it creates a file called Program.cs. We’re not going to do anything with Program.cs yet. I’m going to start by creating a new class file to represent a log entry. I’ll call it NginxLogEntry. I can see in my sample data that we have a date field, so I know that I’m going to need internationalization, owing to the cultural info needed to render a date. So, let’s get the basics in place with a using statement for the globalization package, a namespace, and the class. Visual Studio likes to mark the classes with an internal access modifier. Call me old-fashioned. I always change them to public, assuming that’s appropriate, and in this case, it is:
using System.Globalization;
namespace WebLogReporter
{
public class NginxLogEntry
{
//TODO: the rest of the code will go here
}
}
With the basics out of the way, let’s set up our member variables. Aside from a couple of constructors, that’s really all we’ll need since this class is designed to represent the line entries in the log.
The fields we’re interested in are visually identifiable in the preceding data sample:
ServerIPAddress represents the IP address of the web server from which the log was taken.RequestDateTime represents the date and time of each request in the log.Verb represents the HTTP verb or request method. We’ll be supporting four, though there are many more available.Route represents the route of the request. Our sample is from a RESTful API.ResponseCode represents the HTTP response code for the request. Successful codes are in the 200 and 300 range. Unsuccessful codes are in the 400 and 500 range.SizeInBytes represents the size of the data returned by the request.RequestingAgent represents the HTTP agent used to make the request. This is usually a reference to the web browser used, but in all the cases in our sample, it is a client written in Python 3 using the popular requests library.
In addition to our fields, I’ll start with an enum to store the four acceptable values for the HTTP methods, which I’ve called HTTPVerbs. The rest are represented with straightforward auto-properties:
public enum HTTPVerbs { GET, POST, PUT, DELETE }
public string ServerIPAddress { get; set; }
public DateTime RequestDateTime { get; set; }
public HTTPVerbs Verb { get; set; }
public string Route { get; set; }
public int ResponseCode { get; set; }
public int SizeInBytes { get; set; }
public string RequestingAgent { get; set; }
Now that I’ve got my enumeration and properties in place, I’m going to make a couple of constructors. I want one constructor that allows me to pass in a line from the log. The constructor will parse the line and return a fully populated class with the log line as an input. Here’s the top of the first constructor:
public NginxLogEntry(String LogLine)
{
First, I’ll take the log line being passed in and split it into a string array, using the .Split() method, which is part of the string class:
var parts = LogLine.Split(' ');
While developing, I run into some corner cases. Sometimes, the log lines don’t have 12 fields, as I expect. To account for this, I add a conditional that detects log lines that come in with fewer than 12 parts. This rarely happens but when it does, I want to send them to the console so that I can see what is going on. This is the kind of thing you’d probably take out. I’m embracing my inner stovepipe developer here, so I’m leaving it in:
if(parts.Length < 12)
{
Console.WriteLine(LogLine);
}
Now, let’s set to work taking apart the line based on the split. It’s pretty easy to pick out the server IP address as the first element of the split array:
ServerIPAddress = parts[0];
We don’t care about those two dashes in positions 1 and 2. We can see the date in the third position. Dealing with dates has always been slightly more fun than your average root canal. Think about all the formatting and the parsing needed just to get it into something we know will work with the database code that we’ll eventually write. Thankfully, C# handles this with aplomb. We pull out the date parts and we use a custom date format parser. I don’t really care about expressing the date in terms of locale, so I’ll use InvariantCulture as the second argument in the date parse:
var rawDateTime = parts[3].Split(' ')[0].Substring(1).Trim();
RequestDateTime = DateTime.ParseExact(rawDateTime, "dd/MMM/yyyy:HH:mm:ss", CultureInfo.InvariantCulture);
Next, we get to work parsing the HTTP verb. It needs to conform to the enum we defined at the top of the class. I start by pulling the relevant word and making sure it’s clean by giving it a quick trim. Then, I cast it to the enumeration type. I probably should have used tryParse(), but I didn’t. It still works with the input sample if I don’t, and that’s the kind of thinking that lands us in a stovepipe prison later:
var rawVerb = parts[5].Trim().Substring(1);
Verb = (HTTPVerbs)Enum.Parse(typeof(HTTPVerbs), rawVerb);
The Route value, the ResponseCode value, and the SizeInBytes value are just grabbed based on their position. In the latter two cases, I just used int.parse() to turn them into integers:
Route = parts[6].Trim();
ResponseCode = int.Parse(parts[8].Trim());
SizeInBytes = int.Parse(parts[9].Trim());
Lastly, I need the RequestingAgent. The sample data has some pesky double quotes that I don’t want to capture, so I’ll just use the string.replace() method to replace them with null, effectively getting rid of them:
RequestingAgent = parts[11].Replace("\"", null);
}
I now have a very useful constructor that does my line parsing for me automatically. Nice!
My second constructor is more standard fare. I’d like to create NginxLogEntry by simply passing in all the relevant data elements:
public NginxLogEntry(string serverIPAddress, DateTime
requestDateTime, string verb, string route, int
responseCode, int sizeInBytes, string requestingAgent)
{
RequestDateTime = requestDateTime;
Verb = (HTTPVerbs)Enum.Parse(typeof(HTTPVerbs),
verb);
Route = route;
ResponseCode = responseCode;
SizeInBytes = sizeInBytes;
RequestingAgent = requestingAgent;
}
}
}
This class begins as all do – with property definitions. We have a requirement to store the log data in SQL Server. For this, I created a database on my laptop running SQL Server 2019. If you don’t have any experience with SQL Server, don’t worry. This is the only place it’s mentioned. You don’t need SQL knowledge to work with patterns in this book. I created a new database called WebLogEntries, then created a table that matches my object structure. The Data Definition Language (DDL) to create the table looks as follows:
CREATE TABLE [dbo].[WebLogEntries](
[id] [int] IDENTITY(1,1) NOT NULL,
[ServerIPAddress] [varchar](15) NULL,
[RequestDateTime] [datetime] NULL,
[Verb] [varchar](10) NULL,
[Route] [varchar](255) NULL,
[ResponseCode] [int] NULL,
[SizeInBytes] [int] NULL,
[RequestingAgent] [varchar](255) NULL,
[DateEntered] [datetime] NOT NULL,
CONSTRAINT [PK_WebLogEntries] PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON,
ALLOW_PAGE_LOCKS = ON, OPTIMIZE_FOR_SEQUENTIAL_KEY =
OFF) ON [PRIMARY]
) ON [PRIMARY]
As you can see, I have added the ubiquitous auto-incrementing primary key field, simply called id. I also added a field to track when the record was entered and set its default value to SQL Server’s GETDATE() function, which yields the current date on the server.
Let’s move on to the code that reads and writes data with SQL Server. I think most people would use an Object-Relational Mapper (ORM) such as .NET’s EF for this. I prefer to leverage the control and performance I get from working directly with the database. In my IDE, I’ll create a second class called SQLServerStorage. If you’re following along, don’t forget to add the Systems.Data package via NuGet.
As before, I’ll start with the dependencies:
using System;
using System.Collections.Generic;
using System.Text;
using System.Data.SqlClient;
Next, I’ll set up the class:
namespace WebLogReporter
{
public class SQLServerStorage
{
//TODO: the rest of the code goes here
}
}
Unlike the data class we created earlier, this one is all about the methods. The first method I’ll make stores the data in the database using a direct connection. If you’ve only ever used EF, and you understand SQL (which you should), I highly recommend you try this style and test it for speed against your usual EF-driven code. You’ll see a huge difference, especially at scale. I’ll get off my proverbial soapbox now and get back to creating the StoreLogLine method. It takes in the NginxLogEntry class that we just wrote as its sole input:
public void StoreLogLine(NginxLogEntry entry)
{
Next, let’s connect to the database. I use the using syntax for this. If you’ve not used this before (see what I did there?), it’s very convenient since it handles the timely closure and destruction of whatever you create. In this case, I’m creating a database connection. Even in throwaway code, there are things you just don’t do, such as open a connection to a resource and fail to close it. It’s just so rude! This line sets up my connection. I also recommend a strong database password. As usual, I can hide behind the excuse that it’s throwaway code. At this point, I’ve likely repeated this more times than your local government has told you to wear a mask. And as with your local government, it probably won’t be the last time you hear it:
using (SqlConnection con = new
SqlConnection("Server=Localhost;Database=
WebLogReporter;User
Id=SA;Password=P@ssw0rd;"))
{
Next, I’ll build my Data Manipulation Language (DML) statement for inserting data into the database using the connection we just forged. I’ll use the StringBuilder class, which is part of System.Text:
var sql = new StringBuilder("INSERT INTO
[dbo].[WebLogEntries] (ServerIPAddress,
RequestDateTime, Verb, Route, ResponseCode,
SizeInBytes, RequestingAgent) VALUES (");
sql.Append("'" + entry.ServerIPAddress + "',");
sql.Append("'" + entry.RequestDateTime + "', ");
sql.Append("'" + entry.Verb + "', ");
sql.Append("'" + entry.Route + "', ");
sql.Append(entry.ResponseCode.ToString() + ", ");
sql.Append(entry.SizeInBytes.ToString() + ", ");
sql.Append("'" + entry.RequestingAgent + "')");
Next, let’s open the connection, then execute our SQL statement:
con.Open();
using(SqlCommand cmd = con.CreateCommand())
{
cmd.CommandText = sql.ToString();
cmd.CommandType = System.Data.CommandType.Text;
cmd.ExecuteNonQuery();
}
}
}
Fabulous! Now that we’re writing data, it stands to reason that we should also read it. Otherwise, our class would be really cruddy. Or maybe it wouldn’t be? I’ll let you mull that over while I type out the next method signature:
public List<NginxLogEntry> RetrieveLogLines()
{
The read method is going to return a list of NginxLogEntry instances. This is why we made that second constructor in the NginxLogEntry class earlier. I’ll start by instantiating an empty list to use as the return value. After that I’ll make a really simple SQL statement to read all the records from the database:
var logLines = new List<NginxLogEntry>();
var sql = "SELECT * FROM WebLogEntries";
Using the same using syntax as before, I’ll open a connection and read the records:
using (SqlConnection con = new
SqlConnection("Server=Localhost;Database=
WebLogReporter;User Id=SA;Password=P@ssw0rd;"))
{
SqlCommand cmd = new SqlCommand(sql, con);
con.Open();
SqlDataReader reader = cmd.ExecuteReader();
With the select statement executed, I’ll use a reader to get the data out line by line, and for each record, I’ll instantiate a NginxLogEntry class. Since it’s supposed to be prototype code, I’m relying on the positions in the dataset for data retrieval. This is not uncommon, but it is fairly brittle. A restructuring of the table will break this code later. But it’s throwaway code! See? I told you that you’d hear it again:
while (reader.Read())
{
var serverIPAddress = reader.GetString(1);
var requestDateTime = reader.GetDateTime(2);
var verb = reader.GetString(3);
var route = reader.GetString(4);
var responseCode = reader.GetInt32(5);
var sizeInBytes = reader.GetInt32(6);
var requestingAgent = reader.GetString(7);
var line = new NginxLogEntry(serverIPAddress,
requestDateTime, verb, route,
responseCode, sizeInBytes,
requestingAgent);
Now that I’ve constructed the object using the data from the table, I’ll add it to my logLines list and return the list. The using statement handles the closure of all the database resources that I created along the way:
logLines.Add(line);
}
}
return logLines;
}
}
}
To sum it up, the class has two methods. The first, StoreLogLine, takes an instance of the NginxLogEntry class and converts the data into a SQL statement compatible with our table structure. We then perform the insert operation. Since I used the using syntax to open the connection to the database, that connection is automatically closed when we leave the scope of the method.
The second operation works in reverse. RetrieveLogLines executes a select statement that retrieves all our data from the table and converts it into a list of NginxLogEntry objects. The list is returned at the close of the method.
The last component is the output component. The class is called Report. Its job is to convert the records requested from the database into an HTML table, which is then written to a file.
I’ll set up the class with the dependencies and begin the class with the usual setup:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace WebLogReporter
{
public class Report
{
//TODO: the rest of your code goes here
}
Next, I’ll add the method to generate the report:
public void GenerateReport(string OutputPath)
{
I’ll now use the SQLServerStorage class that we made earlier:
var database = new SQLServerStorage();
var logLines = database.RetrieveLogLines();
I have the data. Now, I’ll use another StringBuilder to generate the HTML. It’s table code because this is absolutely not a book on frontend design:
var output = new
StringBuilder("<html><head><title>Web
Log Report</title></head><body>");
output.Append("<table><tr><th>Request
Date</th><th>Verb</th><th>Route</th>
<th>Code</th><th>Size</th><th>Agent
</th></tr>");
foreach (var logLine in logLines)
{
output.Append("<tr>");
output.Append("<td>" +
logLine.RequestDateTime.ToString() + "</td>");
output.Append("<td>" + logLine.Verb + "</td>");
output.Append("<td>" + logLine.Route + "</td>");
output.Append("<td>" +
logLine.ResponseCode.ToString() + "</td>");
output.Append("<td>" +
logLine.SizeInBytes.ToString() + "</td>");
output.Append("<td>" +
logLine.RequestingAgent.ToString() + "</td>");
output.Append("</tr>");
}
output.Append("</table></body></html>");
Finally, we have a wonderful C# one-liner to output the file so that it’s ready for viewing in your favorite browser:
File.WriteAllText(OutputPath, output.ToString());
}
}
}
It might be ugly, but it works. I’ll say it once again just because I can. It’s throwaway code! One trick I advocate when writing throwaway code is to make it so incredibly ugly that nobody in their right mind would put their name to it. I think I’ve accomplished that here. I just used a string builder to create my HTML. No spaces or formatting. It’s basically a minified HTML file, which is, of course, an intended feature and not at all inspired by laziness.
There’s one last thing to do before we put this baby to bed. We need to edit the Program.cs file Visual Studio created as the project’s entry point. This file glues all the other pieces together. The most recent editions of most C# IDEs generate the entry point for console apps within the Program.cs file. This isn’t new. What is new is the format this file takes. The new format lacks the usual constructor and class setup we’ve seen so far in the classes we created from scratch. Behind the scenes, the compiler is generating those definitions for us, but it makes Program.cs look different from everything else. Rather than present all the usual boilerplate, it’s straight to business.
We’ll start by using the WebLogReporter class that we just created:
using WebLogReporter;
We’ll do a perfunctory and minimal test to make sure the right number of arguments were passed in from the command line. We need a path to the log file and an output path. If you don’t pass in the right number of arguments, we’ll give you a little command-line hint, then exit with a non-zero code, in case this is part of some sequence of automation. I know, it’s throwaway code, but I’m not a barbarian:
if (args.Length < 2)
{
Console.WriteLine("You must supply a path to the log file
you want to parse as well as a path for the output.");
Console.WriteLine(@"For example: WebLogReporter
c:\temp\nginx-sample.log c:\temp\report.html");
Environment.Exit(1);
}
Now, we check whether the log input file exists. If it doesn’t, we alert the user to our disappointment and again exit with non-zero code:
if (!File.Exists(args[0]))
{
Console.WriteLine("The path " + args[0] + " is not a
valid log file.");
Environment.Exit(1);
}
If they make it this far, we’re assuming everything is good and we get to work:
var logFileName = args[0];
var outputFile = args[1];
Console.WriteLine("Processing log: " + logFileName);
int lineCounter = 0;
We instantiate SQLServerStorageClass so we can store our records as we read them in:
var database = new SQLServerStorage();
Now, we open the input log file, and with a foreach loop, we take each line and use our parsing constructor in NginxLogEntry to create an NginxLogEntry object. We then feed that to our database class. If we encounter a line that’s a problem, we write out a message that states where the problem occurred so that we can review it later:
foreach(string line in
System.IO.File.ReadLines(logFileName))
{
lineCounter++;
try
{
var logLine = new NginxLogEntry(line);
database.StoreLogLine(logLine);
}
catch
{
Console.WriteLine("Problem on line " + lineCounter);
}
}
We’ve parsed the log data and written it to the database. All that’s left is to use the Report class to write out our HTML:
var report = new Report();
report.GenerateReport(outputFile);
Console.WriteLine("Processed " + lineCounter + " log
lines.");
To sum up, the Program.cs file contains the main program. The current version of C# allows us to dispense with the usual class definition in the main file for the project.
First, we check to make sure the user entered two arguments. It’s hardly a bulletproof check, but it’s good enough to demo.
Next, after making sure the input log file is a legitimate path, we open the file, read it line by line, and save each line to the database.
Once we’ve read all our lines, we read back the data from the database and convert it to HTML using the report object.
Your program is complete; you demonstrate it to the customer, and they are delighted! A week later, their boss has lunch with your boss, and a new requirement comes in stating the client would now like to support logs from two other web server formats: Apache and IIS. They’d also like to select the output from several different formats, including the following:
- HTML (which we have already delivered)
- Adobe PDFs
- Markdown
- JavaScript Object Notation (JSON)
The purpose of the last format, JSON, is that it allows outside clients to ingest that data into other systems for further analysis, such as capturing trend data over time.
While these concise descriptions of the requirements are hardly what we’d want when building a real extension to our program, they are enough to get you thinking.
What would you do?
Have we built a stovepipe system? If not, is there a chance it might evolve into one? Pause for a moment and think about this before reading further.
I believe we have built a stovepipe system. Here’s why:
- Our classes are all single-purpose and coupled directly to the web server log formats
- Our software is directly coupled with SQL Server
- Our output to HTML is the only possible output, given that we didn’t create an interface or structure to abstract the output operation and format
You might be thinking that the second set of requirements was unknown at the time we created our first release. That’s accurate. You might further defend your idea by stating you are not psychic, and there’s no way you could have known you would need to extend the program per the second set of requirements. You’re right there too. Nobody is prescient. But despite that, you know, if for no other reason than you’ve read this chapter so far, that any successful program must support extension. This is true because you know your first iteration has now begotten a request for a second and that always entails changes and additions to the requirements. We can never know how the requirements will change, but we do know that they will change.
Germany
Slovakia
Canada
Brazil
Singapore
Hungary
Philippines
Mexico
Thailand
Ukraine
Luxembourg
Estonia
Lithuania
Norway
Chile
United States
Great Britain
India
Spain
South Korea
Ecuador
Colombia
Taiwan
Switzerland
Indonesia
Cyprus
Denmark
Finland
Poland
Malta
Czechia
New Zealand
Austria
Turkey
France
Sweden
Italy
Egypt
Belgium
Portugal
Slovenia
Ireland
Romania
Greece
Argentina
Malaysia
South Africa
Netherlands
Bulgaria
Latvia
Australia
Japan
Russia
Filter
