IDS/IPS metrics

It is essential to be familiar with a few key metrics that are often used to describe how capable an IDS/IPS is. An IDS/IPS has two main metric classes: detection accuracy and performance metrics. These metrics are mainly used to compare IDS/IPS, which are also known as IDS/IPS evaluations. We will look at these topics in this section.

Detection accuracy

Every packet or connection analyzed has two possibilities – benign or malicious. Also, there are two possibilities for IDS analysis results – an alert is generated or no alert is generated. So, we end up with four possibilities, as described in the following table. This table is called a confusion matrix and is a valuable way to measure the performance of an IDS in classifying the connections or sessions as benign or malicious.

	Benign	Attack
No alert generated	True negative	False negative
Alert generated	False positive	True positive

Table 1.1 – Intrusion detection confusion matrix

Let’s look at each of these cases:

• True positive (TP): TP is the case when the connection is malicious and the IDS correctly alerts.
• True negative (TN): TN is the case when the connection is benign and the IDS correctly avoids generating an alert. Ideally, the TN rate should be 100%; this means that 100% of benign connections will result in an absence of an alert.
• False positive (FP): FP is the case when the connection is benign and the IDS incorrectly generates an alert. Ideally, the FP rate should be 0%, meaning that the IDS does not generate any alerts for benign connections.
• False negative (FN): FN is the case when the connection is malicious and the IDS incorrectly fails to generate an alert. Ideally, the FN rate should be 0%, meaning that the IDS does not fail to generate an alert for malicious connections.

Now, the metrics used for detection accuracy are as follows:

• True positive rate (TPR): The TPR is calculated as the ratio of accurately detected attacks (TP) to the total number of attacks (TP + FN). Note that the total number of attacks is equal to the number of attacks detected (TP) plus the number of attacks missed (FN).

Ideally, the TPR should be equal to 1, which means FN should be 0; in other words, the IDS alerts on all the attacks.

• False positive rate (FPR): The FPR is calculated as the ratio of wrongly detected attacks (FP) to the total number of benign connections (FP + TN).

Ideally, the FPR should be equal to 0, which means FP should be 0; in other words, the IDS does not alert on any of the benign connections.

• Precision rate (PR): The PR is calculated as the ratio of accurately detected attacks (TP) to the total number of alerts generated (TP + FP).

Ideally, the PR should be equal to 1, which means TP should be non-zero and FP should be 0; in other words, all the alerts generated by the IDS should be for attacks.

The preceding metrics are useful in measuring the detection accuracy of IDS or IPS. The values of these metrics are useful for comparison purposes (to choose one system over another) or for benchmarking purposes (to measure the improvement of a system over time).

Generic versus specific signatures – a discussion

For a signature-based IDS/IPS, these metrics mostly depend on how specific or generic the signatures are. If a signature is too generic, it tends to have a good TPR, but at the same time, the FPR also increases. On the other hand, if a signature is too specific, it will result in low FPRs. However, it will also result in a miss when there is a slight modification to the attack; that is, FN increases.

Performance-related IDS/IPS metrics

The traffic volume on the internet, as well as the traffic volume within any network, has been increasing year by year. Figure 1.6 shows how the traffic volume has increased across the internet over the past several decades.

Figure 1.6 – The traffic volume on the internet

In addition, the network IDS/IPS is typically deployed in key points in the network where it must monitor the traffic to and from the entire network. IDS and IPS must perform at an efficient pace to keep up with the increasing traffic loads.

The complexity of the analysis performed by the IDS affects this rating. More complex analysis (for example, a higher number of signatures to check, or more complex signatures to check) leads to an increase in the IDS processing time for a packet. Typical IDS and IPS have configurations, which include various parameter settings that control their behavior, as well as the total database of signatures to match against. By controlling the configuration, we can control the packet processing time of the IDS/IPS, thereby affecting the throughput that can be sustained.

The following metrics are often used to measure the performance of an IDS/IPS:

• Throughput: This is the maximum amount of network data that can be analyzed by the IDS without packet drops. This is measured in bits per second (or megabits per second or gigabits per second).
• Latency: This metric is only applicable to IPS devices since it works in an inline (not offline, passive) fashion. The network traffic traverses the IPS, and packets are forwarded only after the IPS has evaluated it. This introduces a delay in the network traffic, which is measured by this metric. The higher the latency, the worse the performance of the system. This is typically measured in nanoseconds or microseconds.
• Packets per second: This is the maximum number of packets per second that can be analyzed by the IDS without packet drops. This is measured in the number of packets per second. Not all packets are the same; some packets take more time to be analyzed than others. So, this number has to be measured while maintaining the traffic profile as normal as the IDS would typically analyze.
• Packet drop rate: This is the rate that indicates the number of packets that are dropped by the IDS. This is usually specified in the number of packets per second.
• TCP connections per second: This is the rate of TCP connections that can be analyzed by the IDS. This is measured in connections per second.
• Simultaneous TCP connections: This metric indicates the number of TCP connections that the IDS can analyze simultaneously. To analyze a TCP connection, the IDS needs to maintain the TCP state and other data structures, which consume memory. Subsequently, this metric indirectly measures how much memory capacity the IDS has.

The preceding IDS/IPS metrics are useful for the performance evaluation of the system. In order to enable businesses to operate well as well as to provide protection, IDS and IPS devices must be highly efficient.

IDS/IPS evaluation and comparison

IDS/IPS evaluation is a process that involves a series of tests and/or experiments in order to measure the detection accuracy as well as the performance of the system. DARPA evaluations concentrated on detection accuracy during the early years of IDS evolution. Organizations such as ICSA Labs and NSS Labs conduct a series of tests that measure the detection accuracy as well as performance ratings of IDS.

These evaluations have to be taken with a grain of salt since the results will depend very much on the selection of attacks as well as the selection of traffic profiles. However, these results are still highly beneficial and help companies narrow down the IDS/IPS solutions to be evaluated in their environment with their particular traffic and test conditions.

Next, let us look at one of the challenges faced by IDS/IPS – IDS evasions. This is the scenario where the attacker is able to conduct an attack through the IDS without getting detected.