Before we go into the nitty gritty details of systems programming, we need to set the stage. We need to have a common understanding of a couple of things. For instance, what does the term “systems programming” even mean? What is it for? Who is it for?

Let me get started with a definition.

Systems programming is the programming of systems. That might technically be correct, but I don’t think it helps us move forward.

Let us break it down: what is a system?

That one is easy. We have been building systems for ages, so we understand what we mean by a system.

Let me show you one definition:

A system is a set or arrangement of things that are related or connected so as to form a unity or organic whole. It is a collection of components or parts that interact with each other to function. This term is used in various fields such as physics, biology, computer science, and business management, each with slightly different connotations.

Great. But this definition is a bit broad. We might want to focus on computer science or software development. No problem; there are several definitions to choose from as well:

A system is a collection of software components that interact to perform a specific function or set of functions.

That is a lot better. If we dive into this a bit further, we can distinguish between different groups of systems:

• Software systems: This is an integrated set of software components that work together to carry out a specific function or set of functions. Those components can be a database server, micro-services, and a frontend. Those components form the complete system, such as a CRM system, source control repository system, and others like that.
• Operating systems (OSs): You probably know what an OS is. I think you have seen that term so often that there is a fair chance you didn’t even realize it is a system. But it most definitely is an OS that contains many parts and components, such as drivers, tools, helpers, and logs. Together, they deliver a system you as a user can use to run your software on, independently of the hardware.
• Distributed systems: We often refer to loosely connected components on a network as a distributed system. Each part is isolated from the others, but they must collaborate to achieve something worthwhile. For example, Azure DevOps runs on many different servers in the Azure cloud. All the components run potentially on different servers and machines, and these components can even be running in different parts of the world. However, they work together to form a complete solution for the end user.
• Embedded systems: An embedded system is usually a combination of hardware and software. The components are tightly coupled with each other. Developers usually write the software to match specific specifications so it uses the hardware best. Think, for instance, about the systems in your car. If you have a reasonably recent car, you undoubtedly have an entertainment system on board. The word “system” in “entertainment system” is a bit of a giveaway: it consists of many distinct components. There is very likely a device that can collect electromagnetic waves from the air (we call that a radio). That device is connected to some software that interprets those waves and turns them into an electrical signal to feed the speakers. Next to that, a component shows you, as the user, what you are listening to. I am sure you can find a lot of other systems in your car and probably in your TV, your phone, or your refrigerator.

There are many more examples, but I hope you see that a system always consists of individual components that are not useful on their own but, when combined, deliver a solution to a problem.

But hold on. We are not done yet.

Given these definitions and examples, you might think that the art of systems programming is just the programming of these systems, and you would not be wrong. But that is, in general, not what systems programming means. It most certainly is not what I mean by that.

Very, very roughly, we can divide software into two types:

• User-facing software: This is software written to be used by people. It has a user interface (UI) with buttons, lists, labels, and more. People interact with the software by using various means of input modalities.
• Software-facing software: This is software designed to be used by other software. There are no UIs since we have no users. We could say other components are the users, but when I say users, I mean people. Software-facing software interacts with other components through APIs, RPC (Remote Procedure Calls) calls, file transfer, and many other ways. No humans are involved in this.

It is the second type we are most interested in here in this book – software meant to be used by other software.

When is a system user-facing and when is it not?

It is not always clear when people are the primary users of some code or when other processes are. We could be very rigorous and say that anything with a UI is user-oriented; anything else is systems-oriented. That will make life easier for us if we want a clear definition. However, in the real world, the boundaries tend to blur.

Let me give you an example. Have a look at this Visual Studio Solution:

Figure 0.1: Solution Explorer with the Calculator project

We have a very, very simple solution here. It has a main program called MyAwesomeCalculator that contains the main code. This is the entry point of our app, using the console as the UI. All logic is in the MathFunctions class library. This is where the magic happens.

If we go back to our definition of Systems programming, we could say that writing the MathFunctions class library is part of Systems programming. After all, no user will ever interact with the classes and interfaces in that library. It is the code in MyAwesomeCalculator that actually uses it.

Great! This means writing the MathFunctions library is systems programming! Well, not so fast. We might come to another conclusion if we look at the sequence diagram that explains the flow. Figure 0.2 shows this sequence diagram.

Figure 0.2: Sequence diagram for our calculations

As you can see in Figure 0.2, the user initiates an operation: they want to add up two numbers. They enter it in the UI of the Main class. The Main class then instantiates an instance of the Adder class. After that creation, the Main class calls the AddUp(a,b) method. The result is passed back to the Main class and shown to the user. After all this, we could discard the Adder instance.

Great. Where are the boundaries? If we look at it this way, we could say that the code in Adder and, thus, in the MathFunctions library is immediately tied to user actions. So, it is user-facing code instead of systems-facing code.

I still like to use the question of who is using the code to determine what kind of software we are writing. But apparently, this is not enough. We need to go a bit deeper.

The code in MyAwesomeCalculator and MathFunctions are in separate assemblies. The user interacts with one assembly; the other is accessed through code only. But they can still be seen as one. If we run the application, the runtime creates AppDomain for us.

AppDomain in .NET is different than AppDomain in .NET Framework. The latter had more ways to isolate code from each other. That was nice, but it was a typical Windows feature. That did not translate well to other platforms. So, to make .NET applications run on other platforms, they needed to redesign this. This results in AppDomain being less restrictive than it used to be. Still, AppDomain remains a logical boundary between different processes. Code runs in one app domain and cannot access other app domains directly.

Here, we have another clue: our MyAwesomeCalculator app and the associated MathFunctions assembly all run in the same AppDomain. To the OS, they are one. Since we decided that actual people use the Main method, the same applies to all other pieces of code in that particular AppDomain.

Let’s rewrite our solution a bit. See the following screenshot.

Figure 0.3: Our solution with a worker process

We removed the class library with the code that did all the work. Instead, we created a new project. That project is a worker process. Technically, I should have kept that class library and referenced that, but I wanted to keep things simple.

A worker process is a background process that runs all the time (not technically true, but for now, this is true enough). It just sits there doing nothing. Then, suddenly, something of interest happens, and it comes to life, does its job, and goes to idle mode again.

As shown in Figure 0.4, the sequence diagram in this case is also slightly different.

Figure 0.4: Sequence diagram for the new revised architecture

MyAwesomeCalculator and the MathFunctionServices worker are now independent of each other. They each run in their own AppDomain. When the user wants to perform the calculation, they enter this in the UI, which invokes the service. The Worker class picks up the command, creates an instance of the Adder class, calls the AddUp method, and then calls the MyAwesomeCalculator again with the results.

As you can see, the calls between all classes are synchronous (designated by a line with a solid arrowhead) except for the call between Main and Worker. That is asynchronous (designated by a line and an open arrowhead).

That makes sense; the calculator cannot know whether the command has arrived or the service is listening. It just does a fire-and-forget, crosses its digital fingers, and hopes for the best.

This is more like it. This is genuinely writing software used by other software (I am talking about MathFunctionServices here, not MyAwesomeCalculator).

I have not shown you how the code in Main calls Worker and how the result flows back from Worker to Main. After all, they are in separate app domains. So, they cannot share memory, right? That is correct. I did not show you that. But do not worry. I have a couple of chapters dedicated to this.

It is important to realize that MathFunctionServices does not have a UI in the ordinary sense of the word. No user ever touches this code. It lies there, dormant, until its services are required. If we compare that to the first example, we see the differences. That first example had all code loaded on the user’s demand, and it somehow all responded to the users’ actions.

A better definition

So, if we combine all of this, we can determine that systems programming is the art of writing components that can perform a function or a set of functions but interact only with other components.

That is what this book is all about. We will learn how to write software that is to be consumed by other software. That is a whole other way of looking at software, requirements, design considerations, and more compared to software meant for humans.

Writing software for software means other ways of thinking about communications, performance, memory usage, security, and so on. All those topics are covered here in this book. Now, you might say: “But wait a minute. Software written for users should also keep performance in mind!” You are right, but software communicating with software has unique needs.

Later chapters show how you can achieve the desired performance and explain why this is important. Let us agree that a component, potentially called thousands of times per second, could use more thought about performance than a screen with a button that a user might click once an hour. I am exaggerating here, but I am sure you get the point.

The same applies to memory consumption. I believe we should always write all software with memory consumption in mind. However, a component that gets used frequently by many other systems tends to be much more vulnerable to issues with memory than other software programs.

Performance and memory pressure are essential when we think about writing embedded systems. Embedded software usually runs on very limited hardware, so we have to try and take advantage of every trick in the book to get it running as fast as possible and using as little memory as possible.

As promised, we will spend much time looking at ways to communicate with these types of software.

To me, Systems programming is the purest form of software development. It is all about algorithms, tweaks, and trying out every trick in the book to get the most out of it. systems programming is the major league of software development. When you have this covered, all other software you write will also benefit from your newfound knowledge. What you learn when writing systems software will become second nature, and you will improve your overall software writing skills. Does this sound exciting? Then, let’s get started!

We already run into a problem. You most likely are a C# developer. Maybe you are a VB.Net developer. But no matter what language, you are a .NET developer. After all, that is what this book is about.

Traditionally, Systems programming is done in Assembly, C, and C++. Systems programming has always been the realm of hardcore developers who know the systems they are working on inside out. In the early 50s of the last century, people wrote systems software using switches. A switch in the up position meant a 1, and a switch in the down position meant a 0. These early computers had 8, 16, or even more switches that pointed to the memory address to read or write. Then, 8 switches represented all the bits in a byte for that memory address. Above these switches, there were little lights (no, not LEDS: that invention happened later). Those little lights, if illuminated, meant a 1 in that byte (and a 0 if not illuminated). That way, you could read the contents of that memory address.

Do not worry; that kind of low-level programming is not the topic of this book. If you are interested, there are good remakes of the original Altair 8800 that started a company called Microsoft. You can program that computer in this way: use the switches and lights on the front panel to enter your software. That is how Bill Gates and Paul Allen wrote their first software. But we have other tools at our disposal.

Since systems software relies on efficient, fast, and memory-aware code, people often use programming languages close to the metal. That usually means using language such as machine code – such as the switches I mentioned earlier. Assembly language is another language used, especially in the seventies and eighties of the last century. C and later C++ are other examples of languages that can take advantage of the specifics of the hardware. Most parts of Windows, for instance, are written in C.

However, systems developers do not restrict themselves to low-level languages only. Let me give you an example.

Higher-level languages for systems programming

In 1965, IBM published a manual called PL/I Language Specifications. C28-6571. This relatively obscure title is a fascinating read: it outlines the specifications of the PL/I programming language. PL/I, a sort of abbreviation for Programming Language One, is a higher-level programming language. It contains block structures to allow for recursion, many different datatypes, exception handling, and many other features we take for granted today. It truly was a high-level language. However, they used it to write parts of the early OSs inside IBM. Remember, this was in the sixties when every microsecond counted. Machines were extremely slow compared to modern systems, so they had to utilize every trick in the book to make things work. Yet, a high-level language was considered appropriate. That means there is no reason not to use a high-level language today, especially considering memory profilers’ compiler techniques and advantages.

Kernel mode and user mode

OSs and drivers are usually not built using .NET. The reason for this is that drivers and most parts of the OS run in kernel mode.

The CPU in your computer can run in two modes: kernel or system mode and user mode. User mode is where most of the applications run. The CPU shields the applications from using other memory or process spaces. The CPU protects the applications by placing them in a sandbox. That is precisely what you would want: it would be very undesirable for a program to snoop around in another application’s memory. The processor handles this level of security.

Kernel mode, however, does not have those limitations. Software running in kernel mode is less restricted, controlled, and trusted. That makes sense: parts of an OS should be able to run in all parts of the system, including in the space of other applications.

However, to run in kernel, the compiled code needs to have certain flags set, and the layout of the binaries should be very specific. That is the problem we face. Our C# code relies heavily on the .NET Runtime, and that runtime is not built to be used in Kernel mode. So, even if we could compile our code so that the OS would accept it, it still would not work due to the app not loading the runtime.

There are ways around this. There are ways to pre-compile and include the runtime classes in your binary. Then, you can modify that binary to run in kernel mode. However, the results may vary, and the whole thing would be unreliable. Unreliable code is the exact opposite of what a device driver or OS part should be, so we will not get into this in this book. It’s a hack, not a standard way of working.

Although this book does not deal with kernel-mode apps, I want to give you some insight. Especially since systems programming is usually programming “close to the metal,” so to speak, we are interacting with systems that are running in kernel mode.

Kernel mode is a mode in the CPU. A system can request the CPU to turn on kernel mode. If the code requesting it has the proper privileges, the CPU will do so, thus unlocking parts of the memory previously unavailable. The code does what it needs to do, and then the CPU returns to user mode. Since the code is still in memory doing all sorts of things, it is quite wrong to say an app is a kernel or user-mode app. Some apps can switch the CPU into that state, but the app is almost always running in mixed mode: most of the time, it is in user mode, sometimes kernel mode. Oh, and when I say CPU, I mean logical CPU. This toggling happens on that level, not on the chip itself (but it can also do that).

I have Adobe Creative Cloud installed on my machine. We all know Photoshop, Illustrator, and Premiere, but these apps are meant to be accessed through the Creative Cloud app. This app monitors the system and launches any app you need when you need it. It also updates the background and keeps track of your fonts, files, colors, and other things like that.

Whenever you read something like “runs in the background,” you might expect some systems programming going on, and indeed, there is.

For example, I get this image if I start Performance Monitor on my system and add the % Privileged Time and % User Time counters for the Adobe Desktop Service process.

Figure 0.5: Performance Monitor showing kernel and user times

The red line in Figure 0.5 shows how much time the Adobe Desktop Service spends in user time. The green line, however, shows how long the service is running in privileged time, and privileged time is just a fancy term for kernel time.

As you can see, this app is doing much work in the kernel time. Although I have to admit, I have no clue what it is doing there, but I am sure it is all for a good reason.

We will encounter kernel mode later in other chapters but we will not build apps that run in it.

Why use .NET?

So, we established that we cannot build an OS or a device driver in .NET. That might lead to the question: “Can we use .NET for systems programming?” The answer is a big yes. Otherwise, this would have been a very thin and short book.

Shall we have a look at our recently discovered definition of systems programming? “Writing software used by other software, as a part of a bigger system that works together to achieve a certain goal.” I have shortened the definition, but it is all about this.

Looking at it this way, we can use .NET to write that software. Better yet: I bet .NET is one of the best choices to do so.

.NET offers many advantages over plain C or even C++ (not the managed kind of C++, that is still .NET.)

Back in the day, when we used .NET-Framework-based applications, it would have been a bad idea to use that for systems programming. However, with the introduction of the latest versions of .NET, many disadvantages have been taken care of. With many disadvantages out of the way, .NET-based systems are a viable choice for these kinds of systems.

C and C++ still are excellent languages for low-level systems code. However, C# and .NET Core have their advantages as well.

This table lays out the differences.

Table 0.1: Comparison of C# and C/C++

As you can see, both options have advantages and disadvantages. However, most of the disadvantages of .NET Core can be removed using clever tricks and smart programming. Those are the topics of the rest of this book.

C# is a very mature and well-designed language. The capabilities far exceed what developers had when they used C to build, for example, the Unix OS.

What is .NET anyway?

.NET Core is the next version of the over two decades old framework that was meant to help developers get their work done quickly.

It all started with .NET Framework 1 back in 2002. Microsoft presented it as the end-all solution to many issues developers were facing. Fun fact: the project had the internal code name Project 42. You get bonus points if you know why they chose that name.

In the years following the introduction, we have seen many different functions of .NET Framework. Microsoft released the last version of .NET Framework on April 18, 2019.

Before that, Microsoft realized they needed to support other platforms as well. They wanted .NET to be available everywhere, including Linux, Macintosh, and most mobile devices. That meant they had to make fundamental changes to the runtime and the framework. Instead of having different runtime versions for each platform, they decided to have a unified version. That became .NET Core. Microsoft released this in June 2016.

.NET Standard was a set of specifications. The specifications told all developers which features of the runtime were available in which version of the runtime. Most developers did not understand the purpose of .NET Standard and assumed it was yet another version of the runtime. But once they got the idea behind this, it made a lot of sense. If you need a specific API, look it up in the documentation, see what version of .NET Standard it was supported, and then check whether your desired runtime supported that version of .NET Standard.

An example might be helpful here. Let’s say you build an app that does some fancy drawing on the screen. You have worked with System.Drawing.Bitmap before, so you want to use that again. However, your new app should be running on .NET Core. Can you reuse your code? If you look up the documentation of the System.Drawing.Bitmap class, you see the following:

Table 0.2: Support for System.Drawing.Bitmap

Darn. This class is not part of .NET Standard. It is not available in all runtimes out there. You need to find another way to draw your images.

Your app also communicates with the outside world. It uses the HttpClient class, found in the System.Net.Http namespace. Can you move that to other platforms? Again, we need to look up the documentation of that class. There, we see this table:

Table 0.3: Support for Sstem.Net.Http.HttpClient

Now, that is more like it. HttpClient is part of the .NET Standard specification, which means that all runtimes that support the mentioned versions of .NET Standard implement this class. You are good to go!

.NET, .NET Framework, .NET Standard – what is all this?

Table 0.3 shows .NET Framework, .NET Standard, and .NET but not .NET Core. We do see .NET, though. What is this all about?

.NET Core was introduced to sit next to .NET Framework. Microsoft intended for .NET Framework to support Windows devices. However, as I explained, Microsoft later decided to support other devices, OSs, and other hardware architectures; hence the introduction of .NET Core. Then, they realized that this complicated things a lot. People lost track of what they could use and where they could use it. The solution to this was the introduction of the .NET Standard specifications, but that only worsened things – even the people who were not confused initially lost track of what was going on.

The version numbering was an issue as well. We have .NET Framework version 4.8.1 that matched .NET Standard 2.1. .NET Core 3.1 also supported .NET Standard 2.1. Many people had no idea what was happening. They could not understand why a .NET (Core) version of 3.0 was newer than .NET 4.5.

Microsoft saw this problem as well. They also had internal issues: they had to backport a lot of the code in the libraries so it would be available everywhere. To eliminate this mess once and for all, they announced that .NET Framework 4.8 would be the last version. .NET Core 3.1 would be the last version. From now on, it was all unified in something called .NET. Then, to prevent issues with the numbering, .NET started with the number 5.

They also made it easier to track when new versions would come out. Every single year, there will be a new version of .NET. So far, the odd numbers are under Long Term Support (LTS); the even numbers are under Standard Term Support (STS). STS is 18 months, and LTS is 3 years.

.NET 5 was an STS version, and since it was released in November 2020, the support ended in May 2022. .NET 6 was an LTS version. Released in November 2021, support ends November 2024. .NET 7 is again an STS, released in November 2022, with an end of life in May 2024.

By the time of writing this book, the preview versions of .NET 8 are out, and that will be an LTS version.

This is what I use in this book.

Now, the versioning is clear. The release cycle is understood. We can finally let that go. We can focus on building cool stuff instead of worrying about versions.

Programming languages – a choice to make

We are not done yet. We have figured out which version of the runtime we need. But the runtime is just that: a runtime. A set of libraries that we can use. Those libraries have a lot of tools and pre-built classes available, so we do not have to write that. That is awesome. However, we still have to write some code ourselves. We do that in a programming language, and then link to the libraries, compile the code, and have a binary we can deploy and run.

What language should we use?

Microsoft offers us three choices. Others have made their own .NET-compatible languages, but we ignore them. These days, the main languages to write .NET code are C#, F#, and Visual Basic.

F# is a language used for functional programming. This is a different approach to programming than most people are used to, but the financial domain and data-intensive systems use it a lot.

Visual Basic is an excellent language for people just getting started in development. Back in the nineties, at the end of the last century, it was one of the few options people had to build GUI systems rapidly. When .NET came along, Microsoft quickly ported Visual Basic to support this framework, so developers did not have as steep a learning curve. However, usage of Visual Basic is dwindling now that Microsoft stopped co-evolving it with C#.

C# is the language we use in this book.

Although not coupled with the available runtime, Microsoft seems to release a new version of the language around the same time they release a new version of .NET. Version 11 of the language came out in November 2022. Version 12 of C# is now in preview when writing this book.

Each new version of the language has improvements, but many are syntactic. That means that if you cannot use the latest language version, you can still use all the features in the runtime. They are officially decoupled. Sometimes, it is just a bit more typing work.

The .NET Runtime is an excellent foundation for building all sorts of systems. The ecosystem surrounding .NET is very extensive. Next, a huge group of people contributes to the framework daily. It is hard to think of a task that cannot be performed with .NET or one of the thousands of NuGet packages available.

Again, real Kernel mode systems, such as device drivers, are best built with non-managed languages. However, for all other purposes, .NET and C# are an excellent choice.

Congratulations! You have made the first steps towards becoming a systems programmer. You now know what systems programming is and how it differs from the usual day-to-day programming you might be used to doing.

You know about the background of programming and the challenges our predecessors faced, and you know why .NET is such an awesome tool to build systems software in.

We are ready to take the next step. We will dive into the nitty-gritty details. However, before we do that, we will need to talk about APIs and .NET Framework, its upsides, and its downsides. So, let’s go!

I asked you to follow along. I requested that you open up your development environment and do what I do. However, to do that, you need to set up the right kind of development environment so that you can actually do what I do.

Let me help you with that.

I use Visual Studio 2022 Enterprise. There is no particular reason I use the Enterprise version besides having that on my machine. There are two other versions: The Professional and the free Community edition. All three versions are fine for the things we want to do. However, the Enterprise edition does have some debugging tools we might need when discussing debugging. When that time comes, I will pinpoint the differences and show you other ways of achieving your goals.

Alternatives such as JetBrains Rider and Visual Studio Code also work, but you might have to do more work yourself when we go into performance tuning and debugging. Again, I will tell you about these when we get there.

I have limited experience with Rider, so I cannot tell you precisely what you need to do, but I am sure that when you are an experienced developer, you can translate what I am showing you into the tools you know and love.

Use what you have and what you know. I am cool with that.

If you decide to go with Visual Studio, which I highly recommend, you should use version 2022 instead of 2019. The latest versions of .NET and C# offer a lot related to performance tuning and memory optimizations. Those versions are only available in the 2022 version of Visual Studio. So, make sure you have that one on your device.

Next to that, we will be doing a lot of console stuff. That means using PowerShell: gone are the days of using cmd.exe.

I highly recommend downloading Windows Terminal. With Terminal, you can have all sorts of consoles. We will use PowerShell most of the time, but when we talk about Linux, we will use the WSL feature to use our machines as Linux machines.

Downloading and installing Terminal is a breeze: you can find it on the Microsoft Store.

Make sure to install Windows Subsystem for Linux as well. Instructions on how to do that are all over the internet; I will not repeat that here.

Once you have installed all of your favorite tools, you can select any one of them in your Terminal. Mine looks like this:

Figure 0.6: Windows Terminal with different shells

As you can see, I have PowerShell, Command Prompt, Ubuntu, Azure Cloud Shell, and some more installed. Selecting one of them is a matter of clicking.

Switching between Linux and Windows has never been easier!

Another tool we will be using later on is WinDbg. WinDbg is an extremely powerful external debugger. It can give you a lot of information about the processes you are interested in. It runs standalone, so you do not have to attach Visual Studio to the process. There are versions available for both X86 and ARM, so it is usable on many devices. You can find WinDbg on the Microsoft website at https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/. Download and install that. WinDbg might become one of your latest best friends.

Next, you might want to install PerfView. It is a free and open-source performance monitoring tool from Microsoft, specially built for analyzing performance on .NET applications.

You can find the source code at https://github.com/Microsoft/perfview. You can download the sources and build the tool yourself or grab one of the pre-build versions. Those are also on that same site. I would suggest building yourself and looking through the source code: there are some terrific examples of how to build software like this. I do not intend to describe how the tool works internally, but I will use it when discussing performance.

Now, all you need is a cup of your favorite beverage, and we are good to go!