Chapter 6: Business Process Definition and Exploratory Data Analysis

Activity 13: Carry Out Mapping to Gaussian Distribution of Numeric Features from the Given Data

1. Download the bank.csv. Now, use the following commands to read the data from it:

   import numpy as np
   import pandas as pd
   import seaborn as sns
   import time
   import re
   import os
   import matplotlib.pyplot as plt
   sns.set(style="ticks")
   
   # import libraries required for preprocessing
   import sklearn as sk
   from scipy import stats
   from sklearn import preprocessing
   
   # set the working directory to the following
   os.chdir("/Users/svk/Desktop/packt_exercises")
   
   # read the downloaded input data (marketing data)
   df = pd.read_csv('bank.csv', sep=';')

2. Identify the numeric data from the DataFrame. The data can be categorized according to its type, such as categorical, numeric (float, integer), date, and so on. We identify numeric data here because we can only carry out normalization on numeric data:

   numeric_df = df._get_numeric_data()
   numeric_df.head()

   The output is as follows:

   

   Figure 6.12: DataFrame

3. Carry out a normality test and identify the features that have a non-normal distribution:

   numeric_df_array = np.array(numeric_df) # converting to numpy arrays for more efficient computation
   
   loop_c = -1
   col_for_normalization = list()
   
   for column in numeric_df_array.T:
       loop_c+=1
       x = column
       k2, p = stats.normaltest(x) 
       alpha = 0.001
       print("p = {:g}".format(p))
           
       # rules for printing the normality output
       if p < alpha:
           test_result = "non_normal_distr"
           col_for_normalization.append((loop_c)) # applicable if yeo-johnson is used
           
           #if min(x) > 0: # applicable if box-cox is used
               #col_for_normalization.append((loop_c)) # applicable if box-cox is used
           print("The null hypothesis can be rejected: non-normal distribution")
           
       else:
           test_result = "normal_distr"
           print("The null hypothesis cannot be rejected: normal distribution")

   The output is as follows:

   

   Figure 6.13: Normality test and identify the features

   Note

   The normality test conducted here is based on D'Agostino and Pearson's test (https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.normaltest.html), which combines skew and kurtosis to identify how close the distribution of the features is to a Gaussian distribution. In this test, if the p-value is less than the set alpha value, then the null hypothesis is rejected, and the feature does not have a normal distribution. Here, we look into each column using a loop function and identify the distribution of each feature.

4. Plot the probability density of the features to visually analyze their distribution:

   columns_to_normalize = numeric_df[numeric_df.columns[col_for_normalization]]
   names_col = list(columns_to_normalize)
   
   # density plots of the features to check the normality
   columns_to_normalize.plot.kde(bw_method=3)

   The density plot of the features to check the normality is as follows:

   

   Figure 6.14: Plot of features

   Note

   Multiple variables' density plots are shown in the previous graph. The distribution of the features in the graph can be seen with a high positive kurtosis, which is not a normal distribution.

5. Prepare the power transformation model and carry out transformations on the identified features to convert them to normal distribution based on the box-cox or yeo-johnson method:

   pt = preprocessing.PowerTransformer(method='yeo-johnson', standardize=True, copy=True)
   normalized_columns = pt.fit_transform(columns_to_normalize)
   normalized_columns = pd.DataFrame(normalized_columns, columns=names_col)

   In the previous commands, we prepare the power transformation model and apply it to the data of selected features.

6. Plot the probability density of the features again after the transformations to visually analyze the distribution of the features:

   normalized_columns.plot.kde(bw_method=3)

   The output is as follows:

   

   Figure 6.15: Plot of features