Standardizing your data

For many machine learning algorithms, performance is highly sensitive to the relative scale of features. For that reason, it is often important to standardize your features. To standardize a feature means to shift all of its values so that their mean = 0 and to scale them so that their variance = 1.

One instance when normalizing is useful is when featuring the PE header of a file. The PE header contains extremely large values (for example, the SizeOfInitializedData field) and also very small ones (for example, the number of sections). For certain ML models, such as neural networks, the large discrepancy in magnitude between features can reduce performance.

Getting ready

Preparation for this recipe consists of installing the scikit-learn and pandas packages in pip. Perform the following steps:

pip install sklearn pandas

In addition, you will find a dataset named file_pe_headers.csv in the repository for this recipe.

How to do it...

In the following steps, we utilize scikit-learn's StandardScaler method to standardize our data:

1. Start by importing the required libraries and gathering a dataset, X:

import pandas as pd

data = pd.read_csv("file_pe_headers.csv", sep=",")
X = data.drop(["Name", "Malware"], axis=1).to_numpy()

Dataset X looks as follows:

2. Next, standardize X using a StandardScaler instance:

from sklearn.preprocessing import StandardScaler

X_standardized = StandardScaler().fit_transform(X)

The standardized dataset looks like the following:

How it works...

We begin by reading in our dataset (step 1), which consists of the PE header information for a collection of PE files. These vary greatly, with some columns reaching hundreds of thousands of files, and others staying in the single digits. Consequently, certain models, such as neural networks, will perform poorly on such unstandardized data. In step 2, we instantiate StandardScaler() and then apply it to rescale X using .fit_transform(X). As a result, we obtained a rescaled dataset, whose columns (corresponding to features) have a mean of 0 and a variance of 1.