Much of this book will focus on building deep learning algorithms with libraries in Python, such as TensorFlow and Keras. While these libraries are helpful to build deep neural networks without getting deep into the calculus and linear algebra of deep learning, this chapter will do a deep dive into building a simple neural network in PySpark to make a gender prediction based on height and weight. One of the best ways to understand the foundation of neural networks is to build a model from scratch, without any of the popular deep learning libraries. Once the foundation for a neural network framework is established, understanding and utilizing some of the more popular deep neural network libraries will become much simpler.

dataframes will serve as the framework for any and all data that will be used in building deep learning models. Similar to the pandas library with Python, PySpark has its own built-in functionality to create a dataframe.

There are several ways to create a dataframe in Spark. One common way is by importing a .txt, .csv, or .json file. Another method is to manually enter fields and rows of data into the PySpark dataframe, and while the process can be a bit tedious, it is helpful, especially when dealing with a small dataset. To predict gender based on height and weight, this chapter will build a dataframe manually in PySpark. The dataset used is as follows:

While the dataset...

The dataframe is almost complete; however, there is one issue that requires addressing before building the neural network. Rather than keeping the gender value as a string, it is better to convert the value to a numeric integer for calculation purposes, which will become more evident as this chapter progresses.

This section will require  importing the following:

• from pyspark.sql import functions

This section walks through the steps for the string conversion to a numeric value...

In order to form the building blocks of the neural network, the PySpark dataframe must be converted into an array. Python has a very powerful library, numpy, that makes working with arrays simple.

The numpy library should be already available with the installation of the anaconda3 Python package. However, if for some reason the numpy library is not available, it can be installed using the following command at the terminal:

pip install or sudo pip install will confirm whether the requirements are already satisfied by using the requested library:

import numpy as np

The goal of the neural network that will be developed in this chapter is to predict the gender of an individual if the height and weight are known. A powerful method for understanding the relationship between height, weight, and gender is by visualizing the data points feeding the neural network. This can be done with the popular Python visualization library matplotlib.

As was the case with numpy, matplotlib should be available with the installation of the anaconda3 Python package. However, if for some reason matplotlib is not available, it can be installed using the following command at the terminal:

pip install or sudo pip install will confirm...

The framework in PySpark and the data are now complete. It is time to move on to building the neural network. Regardless of the complexity of the neural network, the development follows a similar path:

1. Input data
2. Add the weights and biases
3. Sum the product of the data and weights

4. Apply an activation function
5. Evaluate the output and compare it to the desired outcome

This section will focus on setting the weights that create the input which feeds into the activation function.

A cursory understanding of the building blocks of a simple neural network is helpful in understanding this section and the rest of the chapter.  Each neural network has...

Neural networks work more efficiently when the inputs are normalized. This minimizes the magnitude of a particular input affecting the overall outcome over other potential inputs that have lower values of magnitude. This section will normalize the height and weight inputs of the current individuals.

The normalization of input values requires obtaining the mean and standard deviation of those values for the final calculation.

This section walks through the steps to normalize the height and weight...

A little bit of validation goes a long way in ensuring that our array is normalized for optimal performance within our upcoming neural network.  

This section will require a bit of numpy magic using the numpy.stack() function.

The following steps walk through validating that our array has been normalized.

1. Execute the following step to print the mean and standard deviation of array inputs:

print('standard deviation')
print(round(X[:,0].std(axis=0),0))
print('mean...