The recent advancements in deep learning can be, to some extent, attributed to the advancements in computing power. The increase in computing power, more specifically the use of GPUs for processing data, has contributed to the leap from shallow neural networks to deeper neural networks. In this chapter, we lay the groundwork for all following chapters by showing you how to set up stable environments for different deep learning frameworks used in this cookbook. There are many open source deep learning frameworks that are used by researchers and in the industry. Each framework has its own benefits and most of them are supported by some big tech company.

By following the steps in this first chapter carefully, you should be able to use local or cloud-based CPUs and GPUs to leverage the recipes in this book. For this book, we've used Jupyter Notebooks to execute all code blocks. These notebooks provide interactive feedback per code block in such a way that it's perfectly suited for storytelling.

The download links in this recipe are intended for an Ubuntu machine or server with a supported NVIDIA GPU. Please change the links and filenames accordingly if needed. You are free to use any other environment, package managers (for example, Docker containers), or versions if needed. However, additional steps may be required. 

Before we get started with training deep learning models, we need to set up our deep learning environment. While it is possible to run deep learning models on CPUs, the speed achieved with GPUs is significantly higher and necessary when running deeper and more complex models.

1. First, you need to check whether you have access to a CUDA-enabled NVIDIA GPU on your local machine. You can check the overview at https://developer.nvidia.com/cuda-gpus.
2. If your GPU is listed on that page, you can continue installing CUDA and cuDNN if you haven't done that already. Follow the steps in the Installing CUDA and cuDNN section.

3. If you don't have access to an NVIDIA GPU on your local machine, you can decide to use a cloud solution. Follow the steps in the Launching a cloud solution section.

Amazon Web Services (AWS) is the most popular cloud solution. If you don't have access to a local GPU or if you prefer to use a server, you can set up an EC2 instance on AWS. In this recipe, we provide steps to launch a GPU-enabled server.

Before we move on with this recipe, we assume that you already have an account on Amazon AWS and that you are familiar with its platform and the accompanying costs.

1. Make sure the region you want to work in gives access to P2 or G3 instances. These instances include NVIDIA K80 GPUs and NVIDIA Tesla M60 GPUs, respectively. The K80 GPU is faster and has more GPU memory than the M60 GPU: 12 GB versus 8 GB. 

2. There are two options when launching an AWS instance. Option 1: You build everything from scratch. Option 2: You use a preconfigured Amazon Machine Image (AMI) from the AWS marketplace. If you choose option 2, you will have to pay additional costs. For an example, see this AMI at https://aws.amazon.com/marketplace/pp/B06VSPXKDX.
3. Amazon provides a detailed and up-to-date overview of steps to launch the deep learning AMI at https://aws.amazon.com/blogs/ai/get-started-with-deep-learning-using-the-aws-deep-learning-ami/.
4. If you want to build the server from scratch, launch a P2 or G3 instance and follow the steps under the Installing CUDA and cuDNN and Installing Anaconda and Libraries recipes.
5. Always make sure you stop the running instances when you're done to prevent unnecessary costs. 

Another popular cloud provider is Google. Its Google Cloud Platform (GCP) is getting more popular and has as a major benefit—it includes a newer GPU type, NVIDIA P100, with 16 GB of GPU memory. In this recipe, we provide the steps to launch a GPU-enabled compute machine.

Before proceeding with this recipe, you should be familiar with GCP and its cost structure.

1. You need to request an increase in the GPU quota before you launch a compute instance with a GPU for the first time. Go to https://console.cloud.google.com/projectselector/iam-admin/quotas.
2. First, select the project you want to use and apply the Metric and Region filters accordingly. The GPU instances should show up as follows:

3. Select the quota you want to change, click on EDIT QUOTAS, and follow the steps.
4. You will get an e-mail confirmation when your quota has been increased.
5. Afterwards, you can create a GPU-enabled machine.
6. When launching a machine, make sure you tick the Allow HTTP traffic and Allow HTTPs traffic boxes if you want to use a Jupyter notebook. 

This part is essential if you want to leverage NVIDIA GPUs for deep learning. The CUDA toolkit is specially designed for GPU-accelerated applications, where the compiler is optimized for using math operations. In addition, the cuDNN library—short for CUDA Deep Neural Network library—is a library that accelerates deep learning routines such as convolutions, pooling, and activation on GPUs.

Make sure you've registered for Nvidia's Accelerated Computing Developer Program at https://developer.nvidia.com/cudnn before starting with this recipe. Only after registration will you have access to the files needed to install the cuDNN library. 

1. We start by downloading NVIDIA with the following command in the terminal (adjust the download link accordingly if needed; make sure you use CUDA 8 and not CUDA 9 for now):

curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb

2. Next, we unpack the file and update all all packages in the package lists. Afterwards, we remove the downloaded file:

sudo dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
sudo apt-get update
rm cuda-repo-ubuntu1604_8.0.61-1_amd64.deb

3. Now, we're ready to install CUDA with the following command:

sudo apt-get install cuda-8-0

4. Next, we need to set the environment variables and add them to the shell script .bashrc:

echo 'export CUDA_HOME=/usr/local/cuda' >> ~/.bashrc
echo 'export PATH=$PATH:$CUDA_HOME/bin' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDA_HOME/lib64' >> ~/.bashrc

5. Make sure to reload the shell script afterwards with the following command:

source ~/.bashrc

6. You can check whether the CUDA 8.0 driver and toolkit are correctly installed using the following commands in your terminal:

nvcc --version
nvidia-smi

The output of the last command should look something like this:

7. Here, we can see that an NVIDIA P100 GPU with 16 GB of memory is correctly connected and ready to use. 

8. We are now ready to install cuDNN. Make sure the NVIDIA cuDNN file is available on the machine, for example, by copying from your local machine to the server if needed. For Google cloud compute engine (make sure you've set up gcloud and the project and zone are set up correctly), you can use the following command (replace local-directory and instance-name with your own settings):

gcloud compute scp local-directory/cudnn-8.0-linux-x64-v6.0.tgz instance-name

9. First we unpack the file before copying to the right directory as root:

cd
tar xzvf cudnn-8.0-linux-x64-v6.0.tgz
sudo cp cuda/lib64/* /usr/local/cuda/lib64/
sudo cp cuda/include/cudnn.h /usr/local/cuda/include/

10. To clean up our space, we can remove the files we've used for installation, as follows:

rm -rf ~/cuda
rm cudnn-8.0-linux-x64-v5.1.tgz

One of the most popular environment managers for Python users is Anaconda. With Anaconda, it's straightforward to set up, switch, and delete environments. Therefore, one can easily run Python 2 and Python 3 on the same machine and switch between different installed versions of installed libraries if needed. In this book, we purely focus on Python 3 and every recipe can be run within one environment: environment-python-deep-learning-cookbook.

1. You can directly download the installation file for Anaconda on your machine as follows (adjust your Anaconda file accordingly):

curl -O https://repo.continuum.io/archive/Anaconda3-4.3.1-Linux-x86_64.sh

2. Next, run the bash script (if necessary, adjust the filename accordingly):

bash Anaconda3-4.3.1-Linux-x86_64.sh

Follow all prompts and choose 'yes' when you're asked to to add the PATH to the .bashrc file (the default is 'no').

3. Afterwards, reload the file:

source ~/.bashrc

4. Now, let's set up an Anaconda environment. Let's start with copying the files from the GitHub repository and opening the directory:

git clone https://github.com/indradenbakker/Python-Deep-Learning-Cookbook-Kit.git
cd Python-Deep-Learning-Cookbook-Kit

5. Create the environment with the following command:

conda env create -f environment-deep-learning-cookbook.yml

6. This creates an environment named environment-deep-learning-cookbook and installs all libraries and dependencies included in the .yml file. All libraries used in this book are included, for example, NumPy, OpenCV, Jupyter, and scikit-learn. 
7. Activate the environment:

source activate environment-deep-learning-cookbook

8. You're now ready to run Python. Follow the next recipe to install Jupyter and the deep learning frameworks used in this book.

As mentioned in the introduction, Jupyter Notebooks have gained a lot of traction in the last couple of years. Notebooks are an intuitive tool for running blocks of code. When creating the Anaconda environment in the Installing Anaconda and Libraries recipe, we included Jupyter in our list of libraries to install. 

1. If you haven't installed Jupyter yet, you can use the following command in your activated Anaconda environment on the server:

 conda install jupyter

2. Next, we move back to the terminal on our local machine.

3. One option is to access the Jupyter Notebook running on a server using SSH-tunnelling. For example, when using Google Cloud Platform:

gcloud compute ssh --ssh-flag="-L 8888:localhost:8888"  --zone "europe-west1-b" "instance-name" 

You're now logged in to the server and port 8888 on your local machine will forward to the server with port 8888.

4. Make sure to activate the correct Anaconda environment before proceeding (adjust the name of your environment accordingly):

source activate environment-deep-learning-cookbook

5. You can create a dedicated directory for your Jupyter notebooks:

mkdir notebooks
cd notebooks

6. You can now start the Jupyter environment as follows:

jupyter notebook

This will start Jupyter Notebook on your server. Next, you can go to your local browser and access the notebook with the link provided after starting the notebook, for example, http://localhost:8888/?token=1fa4e9aea99cd7be2b974557eee3d344ca3c992f5861834f.

One of the most—if not the most—popular frameworks at the moment is TensorFlow. The TensorFlow framework is created, maintained, and used internally by Google. This general open source framework can be used for any numerical computation by using data flow graphs. One of the biggest advantages of using TensorFlow is that you can use the same code and deploy it on your local CPU, cloud GPU, or Android device. TensorFlow can also be used to run your deep learning model across multiple GPUs and CPUs.

1. First, we will show how to install TensorFlow from your terminal (make sure that you adjust the link to the TensorFlow wheel for your platform and Python version accordingly):

pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.3.0-cp35-cp35m-linux_x86_64.whl

This will install the GPU-enabled version of TensorFlow and the correct dependencies.

2. You can now import the TensorFlow library into your Python environment:

import tensorflow as tf

3. To provide a dummy dataset, we will use numpy and the following code:

import numpy as np
x_input = np.array([[1,2,3,4,5]])
y_input = np.array([[10]])

4. When defining a TensorFlow model, you cannot feed the data directly to your model. You should create a placeholder that acts like an entry point for your data feed:

x = tf.placeholder(tf.float32, [None, 5])
y = tf.placeholder(tf.float32, [None, 1])

5. Afterwards, you apply some operations to the placeholder with some variables. For example:

W = tf.Variable(tf.zeros([5, 1]))
b = tf.Variable(tf.zeros([1]))
y_pred = tf.matmul(x, W)+b

6. Next, define a loss function as follows:

loss = tf.reduce_sum(tf.pow((y-y_pred), 2))

7. We need to specify the optimizer and the variable that we want to minimize:

train = tf.train.GradientDescentOptimizer(0.0001).minimize(loss)

8. In TensorFlow, it's important that you initialize all variables. Therefore, we create a variable called init:

init = tf.global_variables_initializer()

We should note that this command doesn't initialize the variables yet; this is done when we run a session.

9. Next, we create a session and run the training for 10 epochs:

sess = tf.Session()
sess.run(init)

for i in range(10):
    feed_dict = {x: x_input, y: y_input}
    sess.run(train, feed_dict=feed_dict)

10. If we also want to extract the costs, we can do so by adding it as follows:

sess = tf.Session()
sess.run(init)

for i in range(10):
    feed_dict = {x: x_input, y: y_input}
    _, loss_value = sess.run([train, loss], feed_dict=feed_dict)
    print(loss_value)

11. If we want to use multiple GPUs, we should specify this explicitly. For example, take this part of code from the TensorFlow documentation:

c = []
for d in ['/gpu:0', '/gpu:1']:
    with tf.device(d):
        a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])
        b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])
c.append(tf.matmul(a, b))
with tf.device('/cpu:0'):
    sum = tf.add_n(c)
# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
# Runs the op.
print(sess.run(sum))

As you can see, this gives a lot of flexibility in how the computations are handled and by which device.

Keras is a deep learning framework that is widely known and adopted by deep learning engineers. It provides a wrapper around the TensorFlow, CNTK, and the Theano frameworks. This wrapper you gives the ability to easily create deep learning models by stacking different types of layers. The power of Keras lies in its simplicity and readability of the code. If you want to use multiple GPUs during training, you need to set the devices in the same way as with TensorFlow.

1. We start by installing Keras on our local Anaconda environment as follows:

conda install -c conda-forge keras 

Make sure your deep learning environment is activated before executing this command.

2. Next, we import keras library into our Python environment:

from keras.models import Sequential
from keras.layers import Dense

This command outputs the backend used by Keras. By default, the TensorFlow framework is used:

3. To provide a dummy dataset, we will use numpy and the following code:

import numpy as np
x_input = np.array([[1,2,3,4,5]])
y_input = np.array([[10]])

4. When using sequential mode, it's straightforward to stack multiple layers in Keras. In this example, we use one hidden layer with 32 units and an output layer with one unit:

model = Sequential()
model.add(Dense(units=32, input_dim=x_input.shape[1]))
model.add(Dense(units=1))

5. Next, we need to compile our model. While compiling, we can set different settings such as loss function, optimizer, and metrics:

model.compile(loss='mse',
              optimizer='sgd',
              metrics=['accuracy'])

6. In Keras, you can easily print a summary of your model. It will also show the number of parameters within the defined model:

model.summary()

In the following figure, you can see the model summary of our build model:

7. Training the model is straightforward with one command, while simultaneously saving the results to a variable called history:

history = model.fit(x_input, y_input, epochs=10, batch_size=32)

8. For testing, the prediction function can be used after training:

pred = model.predict(x_input, batch_size=128)

PyTorch is the Python deep learning framework and it's getting a lot of traction lately. PyTorch is the Python implementation of Torch, which uses Lua. It is backed by Facebook and is fast thanks to GPU-accelerated tensor computations. A huge benefit of using PyTorch over other frameworks is that graphs are created on the fly and are not static. This means networks are dynamic and you can adjust your network without having to start over again. As a result, the graph that is created on the fly can be different for each example. PyTorch supports multiple GPUs and you can manually set which computation needs to be performed on which device (CPU or GPU).

1. First, we install PyTorch in our Anaconda environment, as follows:

conda install pytorch torchvision cuda80 -c soumith

If you want to install PyTorch on another platform, you can have a look at the PyTorch website for clear guidance: http://pytorch.org/.

2. Let's import PyTorch into our Python environment:

import torch

3. While Keras provides higher-level abstraction for building neural networks, PyTorch has this feature built in. This means one can build with higher-level building blocks or can even build the forward and backward pass manually. In this introduction, we will use the higher-level abstraction. First, we need to set the size of our random training data:

 batch_size = 32
 input_shape = 5
 output_shape = 10

4. To make use of GPUs, we will cast the tensors as follows:

torch.set_default_tensor_type('torch.cuda.FloatTensor') 

This ensures that all computations will use the attached GPU. 

5. We can use this to generate random training data:

from torch.autograd import Variable
X = Variable(torch.randn(batch_size, input_shape))
y = Variable(torch.randn(batch_size, output_shape), requires_grad=False)

6. We will use a simple neural network having one hidden layer with 32 units and an output layer:

model = torch.nn.Sequential(
          torch.nn.Linear(input_shape, 32),
          torch.nn.Linear(32, output_shape),
        ).cuda()

We use the .cuda() extension to make sure the model runs on the GPU. 

7. Next, we define the MSE loss function:

loss_function = torch.nn.MSELoss()

8. We are now ready to start training our model for 10 epochs with the following code:

learning_rate = 0.001
for i in range(10):
    y_pred = model(x)
    loss = loss_function(y_pred, y)
    print(loss.data[0])
    # Zero gradients
    model.zero_grad()
    loss.backward()

    # Update weights
    for param in model.parameters():
        param.data -= learning_rate * param.grad.data

Microsoft also introduced its open source deep learning framework not too long ago: Microsoft Cognitive Toolkit. This framework is better known as CNTK. CNTK is written in C++ for performance reasons and has a Python API. CNTK supports GPUs and multi-GPU usage. 

1. First, we install CNTK with pip as follows:

pip install https://cntk.ai/PythonWheel/GPU/cntk-2.2-cp35-cp35m-linux_x86_64.whl

Adjust the wheel file if necessary (see https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-Linux-Python?tabs=cntkpy22). 

2. After installing CNTK, we can import it into our Python environment:

import cntk

3. Let's create some simple dummy data that we can use for training:

import numpy as np
x_input = np.array([[1,2,3,4,5]], np.float32)
y_input = np.array([[10]], np.float32)

4. Next, we need to define the placeholders for the input data:

X = cntk.input_variable(5, np.float32)
y = cntk.input_variable(1, np.float32)

5. With CNTK, it's straightforward to stack multiple layers. We stack a dense layer with 32 inputs on top of an output layer with 1 output:

from cntk.layers import Dense, Sequential
model = Sequential([Dense(32),
                Dense(1)])(X)

6. Next, we define the loss function:

loss = cntk.squared_error(model, y)

7. Now, we are ready to finalize our model with an optimizer:

learning_rate = 0.001
trainer = cntk.Trainer(model, (loss), cntk.adagrad(model.parameters, learning_rate))

8. Finally, we can train our model as follows:

for epoch in range(10):
        trainer.train_minibatch({X: x_input, y: y_input})

The MXNet deep learning framework allows you to build efficient deep learning models in Python. Next to Python, it also let you build models in popular languages as R, Scala, and Julia. Apache MXNet is supported by Amazon and Baidu, amongst others. MXNet has proven to be fast in benchmarks and it supports GPU and multi-GPU usages. By using lazy evaluation, MXNet is able to automatically execute operations in parallel. Furthermore, the MXNet frameworks uses a symbolic interface, called Symbol. This simplifies building neural network architectures.

1. To install MXNet on Ubuntu with GPU support, we can use the following command in the terminal:

pip install mxnet-cu80==0.11.0

For other platforms and non-GPU support, have a look at https://mxnet.incubator.apache.org/get_started/install.html.

2. Next, we are ready to import mxnet in our Python environment:

import mxnet as mx

3. We create some simple dummy data that we assign to the GPU and CPU:

import numpy as np
x_input = mx.nd.empty((1, 5), mx.gpu())
x_input[:] = np.array([[1,2,3,4,5]], np.float32)

y_input = mx.nd.empty((1, 5), mx.cpu())
y_input[:] = np.array([[10, 15, 20, 22.5, 25]], np.float32)

4. We can easily copy and adjust the data. Where possible MXNet will automatically execute operations in parallel:

x_input
w_input = x_input
z_input = x_input.copyto(mx.cpu())
x_input += 1
w_input /= 2
z_input *= 2

5. We can print the output as follows:

print(x_input.asnumpy())
print(w_input.asnumpy())
print(z_input.asnumpy())

6. If we want to feed our data to a model, we should create an iterator first:

batch_size = 1
train_iter = mx.io.NDArrayIter(x_input, y_input, batch_size, shuffle=True, data_name='input', label_name='target')

7. Next, we can create the symbols for our model:

X = mx.sym.Variable('input')
Y = mx.symbol.Variable('target')
fc1 = mx.sym.FullyConnected(data=X, name='fc1', num_hidden = 5)
lin_reg = mx.sym.LinearRegressionOutput(data=fc1, label=Y, name="lin_reg")

8. Before we can start training, we need to define our model:

model = mx.mod.Module(
    symbol = lin_reg,
    data_names=['input'], 
    label_names = ['target']
)

9. Let's start training:

model.fit(train_iter,
            optimizer_params={'learning_rate':0.01, 'momentum': 0.9},
            num_epoch=100,
            batch_end_callback = mx.callback.Speedometer(batch_size, 2))

10. To use the trained model for prediction we:

model.predict(train_iter).asnumpy()

The newest addition to the broad range of deep learning frameworks is Gluon. Gluon is recently launched by AWS and Microsoft to provide an API with simple, easy-to-understand code without the loss of performance. Gluon is already included in the latest release of MXNet and will be available in future releases of CNTK (and other frameworks). Just like Keras, Gluon is a wrapper around other deep learning frameworks. The main difference between Keras and Gluon, is that Gluon will (at first) focus on imperative frameworks. 

1. At the moment, gluon is included in the latest release of MXNet (follow the steps in Building efficient models with MXNet to install MXNet). 
2. After installing, we can directly import gluon as follows:

from mxnet import gluon

3. Next, we create some dummy data. For this we need the data to be in MXNet's NDArray or Symbol:

import mxnet as mx
import numpy as np
x_input = mx.nd.empty((1, 5), mx.gpu())
x_input[:] = np.array([[1,2,3,4,5]], np.float32)

y_input = mx.nd.empty((1, 5), mx.gpu())
y_input[:] = np.array([[10, 15, 20, 22.5, 25]], np.float32)

4. With Gluon, it's really straightforward to build a neural network by stacking layers:

net = gluon.nn.Sequential()
with net.name_scope():
    net.add(gluon.nn.Dense(16, activation="relu"))
    net.add(gluon.nn.Dense(len(y_input)))

5. Next, we initialize the parameters and we store these on our GPU as follows:

net.collect_params().initialize(mx.init.Normal(), ctx=mx.gpu())

6. With the following code we set the loss function and the optimizer:

softmax_cross_entropy = gluon.loss.SoftmaxCrossEntropyLoss()
trainer = gluon.Trainer(net.collect_params(), 'adam', {'learning_rate': .1})

7. We're ready to start training or model:

n_epochs = 10

for e in range(n_epochs):
    for i in range(len(x_input)):
        input = x_input[i]
        target = y_input[i]
        with mx.autograd.record():
            output = net(input)
            loss = softmax_cross_entropy(output, target)
            loss.backward()
        trainer.step(input.shape[0])