Here, we'll put the basics of neural network into practice. We'll adapt the logistic regression TenserFlow code into a single hidden layer of neurons. Then, you'll learn the idea behind backpropagation to compute the weights, that is, train the net. Finally, you'll train your first true neural network in TensorFlow.

The TensorFlow code for this section should look familiar. It's just a slightly evolved version of the logistic regression code. Let's look at how to add a hidden layer of neurons that will compute nonlinear combinations of our input pixels.

You should start with a fresh Python session, execute the code to read in, and set up the data as in the logistic model. It's the same code, just copied to the new file:

import tensorflow as tf
import numpy as np
import math
from tqdm import tqdm
%autoindent
try:
    from tqdm import tqdm
except ImportError:
    def tqdm(x, *args, **kwargs):
        return x

You can always go back...

In this section, we'll carefully look at the model we built. First, we'll verify the overall accuracy of our model, then we'll see where the model goes wrong. Finally, we'll visualize the weights associated with several neurons to see what they're looking for:

plt.figure(figsize=(6, 6))
plt.plot(train_acc,'bo')
plt.plot(test_acc,'rx')

Make sure that you've trained your model as we did in the previous section, if not, you might want to stop here and do that first. Because we evaluated our model accuracy every 10 training epochs and saved the result, it's now easy to explore how our model has evolved.

Using Matplotlib, we can plot both the training accuracy (the blue dots) and testing accuracy (the red dots) on the same figure:

Again, if you don't have Matplotlib, that's okay. You can just look at the array values themselves. Note that the training accuracy (blue in color) is usually a little better...

In this section, we'll show you how to build even more complex models with additional hidden layers. We'll adapt our single hidden layer model into a multilayer model known as a deep neural network. Then, we'll discuss choosing how many neurons and layers to use. Finally, we'll train the model itself, being patient, as this might take a while to compute.

Remember when we added a hidden layer of neurons to our logistic regression model? Well, we can do that again, adding another layer to our single hidden layer model. Once you have more than one layer of neurons, we call this a deep neural network. However, everything you learned before can be applied now. As in the previous sections of this chapter, you should make a fresh Python session and execute the code up to num_hidden1 in this section's code file. Then the fun starts.

Now, we'll look into what's going on inside a deep neural network. First, we'll verify the model accuracy. Then, we'll visualize and study the pixel weights. Finally, we'll look at the output weights as well.

After you've trained your deep neural network, let's take a look at the model accuracy. We'll do this the same way that we did for the single hidden layer model. The only difference this time, is that we have many more saved samples of the training and testing accuracy, having gone from many more epochs.

As always, don't worry if you don't have Matplotlib; printing parts of the arrays is fine.

Understanding the multiple hidden layers graph

Execute the following code to see the result:

# Plot the accuracy curves
plt.figure(figsize=(6,6))
plt.plot(train_acc,'bo')
plt.plot(test_acc,'rx')

From the preceding output graph, we reach about 68 percent training accuracy and maybe 63 percent validation...

In this chapter, we embraced deep learning with TensorFlow. Though we started with the simple model of one hidden layer of neurons, it didn't take you long to develop and train a deep neural network for the font classification problem.

You learned about the single and multiple hidden layer model and understood those in detail. You also understand the different types of neural networks and built and trained our first neural network with TensorFlow.

In the next chapter, we'll prove our model with convolutional neural networks, a powerful tool for image classification.