The state of the art in IDS/IPS

The intrusion detection and prevention field has been evolving for a few decades. During this period, several commercial and open source IDS/IPS have been developed. As the nature of the internet and its protocols, as well as the complexity of threats, evolved, the IDS/IPS also had to evolve in order to keep up with the threats. Snort is an open source IDS/IPS that was created in 1998, and over the past 20+ years, it has evolved into one of the leading IDS/IPS software. Bro is another open source project, which started in 1994 and was mainly used in an academic setting for several years. Recently, it was renamed Zeek, and a community has formed around the open source project. Suricata is a relatively late player in the game and was created in 2009. It is a signature-based IDS/IPS similar to Snort. The rule syntax for Suricata is very similar to that of Snort. In addition to the rules, Suricata has many other similarities to Snort in functionality – although the design and implementation are completely different.

These three open source IDS/IPS have kept up with the challenges that they faced and stood the test of time. It may be said that the current state of these three IDS/IPS represents the state of the art in IDS/IPS. In this section, let us describe some of the challenges that these systems have faced and what features solved them.

Stateful analysis

Stateful analysis of the various network protocols is a necessary feature in any IDS/IPS. Snort was completely stateless and basically a packet analysis IDS in the initial years. Even when stateful analysis was introduced in the subsequent years, it was incomplete and insufficient. Ideally, the IDS/IPS must analyze the network traffic exactly as the end hosts would analyze it. This means that the IDS would need to maintain a very similar state to the end hosts. This is not a trivial task. This is the reason why it took decades for Snort to improve its stateful analysis functionality. Currently, one could say that Snort is a stateful IDS/IPS device, even though there are still limitations.

Fast packet acquisition

Historically, IDS/IPS devices used the packet capture library called libpcap. This is a library used by the tcpdump project and was available as open source. libpcap worked great, but as the internet speed increased, this library started becoming a performance bottleneck. In the case of libpcap, the packet data (network traffic) had to be copied several times before reaching the IDS for processing, and this was one of the reasons for the performance issue. Currently, the state of the art uses zero-copy mechanisms in order to improve performance. Although Snort still supports and offers libpcap-based packet acquisition, it offers all the latest packet acquisition mechanisms to be used.

Parallel processing

The state of the art is for the IDS/IPS to perform network traffic analysis using parallel processing – this could be a multi-process-based or multi-thread-based design. Snort started as a single-threaded, single-process IDS, and then evolved into a multi-process design. Currently, Snort uses a multi-threaded design.

In a multi-process and multi-threaded design, an incoming session would be processed by one of the processes or threads. Once a session is analyzed by a process or thread, then all the subsequent network packets for that session will be analyzed by that process or thread. This is called session pinning. Typically, such pinning is based on a hashing approach, where the hash will be based on the source and destination IP addresses, port, and protocol. However, in this approach, two related sessions that hash to two separate processes or threads will result in a lesser-grade analysis.

Pattern matching

Pattern matching has been and is still one of the most important features of IDS/IPS. A single signature may contain several pattern matches. Originally, these were evaluated one rule at a time, one pattern at a time. With time, multi-pattern search algorithms were used in order to speed up the rule processing.

In addition, as opposed to the crude pattern matching of the past, current IDS/IPS devices perform the pattern matching with context. For example, when a pattern is specified, it can also be specified what data to match against – HTTP URI, HTTP header, and so on. This improves the performance since the pattern search can be limited to specific data, and it also improves detection accuracy.

Extending rule language

Most IDS/IPS have a rich rule language. However, there will always be cases that cannot be covered by the limited capability offered by the rule language. Each system – Snort, Suricata, and Zeek – has its own approach to this challenge. Zeek from the Bro days had a full-fledged language to write detections in. So, the challenge really did not apply to Zeek. Snort came up with shared object (SO) rules, whereby custom C code could be written for a particular functionality and released as .so files in a release. Suricata integrated Lua scripting as part of the rule language extension.

App and protocol identification

Historically, Snort rules were based on protocol and port. For example, the rule would specify that it applies to TCP and on ports 80, 8080, and 3128. The list of ports could be more extensive to cover the usual HTTP ports. However, if there is an HTTP session on port 1000, the rule will not be applied against that session. This challenge was solved by introducing the app and protocol identification feature, which is a state-of-the-art feature. All leading IDS/IPS detect the various protocols on any random port to perform the analysis correctly.

File analysis

In certain cases, the IDS/IPS must analyze the data not as a stream of bytes but as a file. This feature is also the state of the art and is part of all leading IDS/IPS.

These standard features represent the state of the art in IDS/IPS. The intent of this discussion was not to present a comprehensive set of features but to give an idea of the various features.

The next section discusses the various metrics that are used to evaluate IDS/IPS. These metrics try to measure how effective the IDS/IPS are from an accuracy perspective, as well as how efficiently they do their tasks.