There are three key parts to most machine learning classifiers, which are as follows:

• The training pipeline
• The neural network setup and training outputs
• The usage pipeline

The training pipeline obtains data, stages it, cleanses it, homogenizes it, and puts it in a format acceptable to the neural network. Do not be surprised if the training pipeline takes 80% to 85% of your effort initially—this is the reality of most machine learning work. Generally, the more realistic the training data, the more time spent on the training pipeline. In enterprise settings, the training pipeline might be an ongoing effort being enhanced perpetually. This is especially true as datasets get larger.

The second part, the neural network setup, and training, can be quick for routine problems and can be a research-grade effort for harder problems. You may find yourself making small...

Machine learning requires training data—often a lot of training data. One of the great things about machine learning is the availability of standard training datasets. These are often used to benchmark node models and configurations and provide a consistent yardstick to gauge performance against previous progress. Many of the datasets are also used in annual global competitions.

This chapter uses training data, which is kindly provided by Yaroslav Bulatov, a machine learning researcher.

You should start by downloading the training data from the following links:

• http://yaroslavvb.com/upload/notMNIST/notMNIST_small.tar.gz
• http://yaroslavvb.com/upload/notMNIST/notMNIST_large.tar.gz

We will download this programmatically, but we should start with a manual download just to peek at the data and structure of the archive. This will be important when we write the pipeline, as we'll need to understand the structure so we can manipulate the data.

The small set is ideal for peeking. You can do this via the following command line, or just use a browser to download the file with an unarchiver to extract the files (I suggest getting familiarized with the command line as all of this needs to be automated):

cd ~/workdir
wget http://yaroslavvb.com/upload/notMNIST/notMNIST_small.tar.gz
tar xvf notMNIST_small.tar.gz

The preceding command line will...

The classifier we're writing seeks to assign unknown images to a class. Classes can be of the following types:

• Feline versus canine
• Two versus seven
• Tumor versus normal
• Smiling versus frowning

In our case, we are considering each letter a class for a total of 10 classes. The training set will reveal 10 subfolders with thousands of images underneath each subfolder. The name of the subfolder is important as it is the label for each of the images. These details will be used by the pipeline to prepare data for TensorFlow.

Ideally, we will want the entire process automated. This way, we can easily run the process end to end on any computer we use without having to...

The next part will focus on image processing and manipulation. This requires some extra libraries you may not have. At this point, it may make sense to just install all the typical packages required in scientific computing, which can be done as follows:

sudo apt-get install python-numpy python-scipy python-matplotlib 
ipython ipython-notebook python-pandas python-sympy python-nose

Additionally, install the image processing library, some external matrix mathematics libraries, and underlying requirements, which can be done as follows:

sudo pip install ndimage
sudo apt-get install libatlas3-base-dev gcc gfortran g++

Much of machine learning is just operations on matrices. We will...

Downloading our training file took a long time. Even extracting all the images took a while. To avoid repeating all this, we will try to do all the work just once and then create pickle files—archives of the Python data structures.

The following procedure runs through each class in our training and test set and creates a separate pickle file for each. In future runs, we'll just begin from here:

 def makePickle(imgSrcPath): 
    data_folders = [os.path.join(tst_files, d) for d in 
     os.listdir(tst_files) if os.path.isdir(os.path.join(tst_files, 
     d))] 
    dataset_names = [] 
    for folder in data_folders: 
        set_filename = folder + '.pickle' 
        dataset_names.append(set_filename) 
        print('Pickling %s.' % set_filename) 
        dataset = loadClass(folder) 
        try: 
            with open(set_filename...

We just created nice, clean, pickle files with preprocessed images to train and test our classifier. However, we've ended up with 20 pickle files. There are two problems with this. First, we have too many files to keep track of easily. Secondly, we've only completed part of our pipeline, where we've processed our image sets but have not prepared a TensorFlow consumable file.

Now we will need to create our three major sets—the training set, the validation set, and the test set. The training set will be used to nudge our classifier, while the validation set will be used to gauge progress on each iteration. The test set will be kept secret until the end of the training, at which point, it will be used to test how well we've trained the model.

The code to do all this is long, so we'll leave you to review the Git repository...

Now, we arrive at the fun part—the neural network. The complete code to train this model is available at the following link: https://github.com/mlwithtf/mlwithtf/blob/master/chapter_02/training.py

To train the model, we'll import several more modules:

 import sys, os
 import tensorflow as tf
 import numpy as np
 sys.path.append(os.path.realpath('..'))
 import data_utils
 import logmanager 

Then, we will define a few parameters for the training process:

 batch_size = 128
 num_steps = 10000
 learning_rate = 0.3
 data_showing_step = 500

After that, we will use the data_utils package to load the dataset that was downloaded in the previous section:

 dataset, image_size, num_of_classes, num_of_channels =  
 data_utils.prepare_not_mnist_dataset(root_dir="..")
 dataset = data_utils.reformat(dataset, image_size, num_of_channels,   
 num_of_classes)
 print...

To save variables from the TensorFlow session for future use, you can use the Saver() function, which is as follows:

 saver = tf.train.Saver() 

Later, you can retrieve the state of the model and avoid tedious retraining by restoring the following checkpoint:

 ckpt = tf.train.get_checkpoint_state(FLAGS.checkpoint_dir) 
 if ckpt and ckpt.model_checkpoint_path: 
 saver.restore(sess, ckpt.model_checkpoint_path) 

Notice how we did not use the test set until the last step. Why not? This is a pretty important detail to ensure that the test remains a good one. As we iterate through the training set and nudge our classifier one way or another, we can sometimes wrap the classifier around the images or overtrain. This happens when you learn the training set rather than learn the features inside each of the classes.

When we overtrain, our accuracy on the iterative rounds of the training set will look promising, but that is all false hope. Having a never-before-seen test set should introduce reality back into the process. Great accuracy on the training set followed by poor results on the test set suggests overfitting.

This is why we've kept a separate test set. It helps indicate the real accuracy of our classifier. This is also why you should never shuffle your dataset...

We will demonstrate the usage of the classifier with notMNIST_small.tar.gz, which becomes the test set. For ongoing use of the classifier, you can source your own images and run them through a similar pipeline to test, not train.

You can create some 28x28 images yourself and place them into the test set for evaluation. You will be pleasantly surprised!

The practical issue with field usage is the heterogeneity of images in the wild. You may need to find images, crop them, downscale them, or perform a dozen other transformations. This all falls into the usage pipeline, which we discussed earlier.

Another technique to cover larger images, such as finding a letter on a page-sized image, is to slide a small window across the large image and feed every subsection of the image through the classifier.

We'll be taking our models into production in future chapters...

Notice how we achieved 86% accuracy. This is a great result for two hours of work, but we can do much better. Much of the future potential is in changing the neural network. Our preceding application used a fully-connected setup, where each node on a layer is connected to each node on the previous layer and looks like this:

As you will learn with more complex network setups in coming chapters, this setup is fast but not ideal. The biggest issue is the large number of parameters, which can cause overfitting of the model on the training data.

You should have learned these skills in the chapter:

• Preparing training and test data
• Creating a training set consumable by TensorFlow
• Setting up a basic neural network graph
• Training the TensorFlow classifier
• Validating the classifier
• Piping in real-world data

Superb progress! We just built a handwriting classifier that would have been world class a decade ago. Also, we built an entire pipeline around the process to fully automate the training setup and execution. This means that our program can be migrated to almost any server and continue to function almost turn-key.